Imaging device and imaging method

ABSTRACT

An imaging device includes a moving image special effect image processing unit that performs image processing of applying a first special effect causing a visual effect over a plurality of frames corresponding to image data of a moving image, and image processing of applying a second special effect corresponding to the first special effect to a still image captured during moving image capturing using image data of a moving image captured before the still image, and a control unit that controls an application form of the first and second special effects in the moving image special effect image processing unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2011-220542 and Japanese PatentApplication No. 2011-220543, filed on Oct. 4, 2011, the entire contentsof which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging device that images a subjectand generates image data of the subject, and an imaging method.

2. Description of the Related Art

In recent years, imaging devices such as digital cameras have been knownthat have not only a function of generating a clear image with a naturalimpression but also a function of intentionally adding an effect such asnoise, shading, and gradation. The imaging device having theabove-mentioned functions can perform shooting giving a specialimpression which has not been provided in the past.

For example, Japanese Laid-open Patent Publication No. 2010-62836discloses a technique of generating an image of a high contrast with agranular feeling (noise feeling) like a film. According to thistechnique, it is possible to capture an image having a rough and dynamicimpression.

Further, Japanese Laid-open Patent Publication No. 2010-74244 disclosesa technique of generating an image whose edge is dimmed. According tothis technique, it is possible to capture an image of an impression thatcapturing is performed by a toy camera.

The above-mentioned techniques of the related arts can be applied tomoving image capturing as well as still image capturing. When movingimage capturing is performed, a unique video expression of a movingimage which is not obtained in a still image can be made using atemporal change in an image.

SUMMARY OF THE INVENTION

An imaging device according to the present invention captures a subject,generates image data of the subject, captures a moving image, andcaptures a still image during moving image capturing including directlyafter moving image capturing end, the device including: a moving imagespecial effect image processing unit that performs image processing ofapplying a first special effect causing a visual effect over a pluralityof frames corresponding to image data of a moving image, and imageprocessing of applying a second special effect corresponding to thefirst special effect to a still image captured during moving imagecapturing using image data of a moving image captured before the stillimage; and a control unit that controls an application form of the firstand second special effects in the moving image special effect imageprocessing unit.

An imaging device according to the present invention captures a subject,generates image data of the subject, captures a moving image, andcaptures a still image during moving image capturing including directlyafter moving image capturing end, the device including: a moving imagespecial effect image processing unit that performs moving image specialeffect image processing of synthesizing captured image data with aplurality of image data captured before the image data at apredetermined ratio; and a control unit that changes the number of imagedata synthesized by the moving image special effect image processingunit depending on whether the captured image data is a moving image or astill image.

An imaging method according to the present invention is performed by animaging device that captures a subject, generates image data of thesubject, captures a moving image, and captures a still image duringmoving image capturing including directly after moving image capturingend, the method including: performing image processing of applying afirst special effect causing a visual effect over a plurality of framescorresponding to image data of a moving image, and image processing ofapplying a second special effect corresponding to the first specialeffect to a still image captured during moving image capturing usingimage data of a moving image captured before the still image.

An imaging method according to the present invention is performed by animaging device that captures a subject, generates image data of thesubject, captures a moving image, and captures a still image duringmoving image capturing including directly after moving image capturingend, the method including: performing moving image special effect imageprocessing of synthesizing captured image data with a plurality of imagedata captured before the image data at a predetermined ratio; andchanging the number of image data synthesized by the moving imagespecial effect image processing unit depending on whether the capturedimage data is a moving image or a still image.

An imaging device according to the present invention captures a subject,generates image data of the subject, and captures a moving image and astill image, the device including: a moving image special effect imageprocessing unit that applies an expression of temporally changing avisual effect over a plurality of frames; and an artistic effectprocessing unit that applies an artistic effect causing a visual effectto one image data, wherein the moving image special effect imageprocessing unit performs transit processing of steadily changing anartistic effect over a plurality of frames, the transit processingsynthesizing an image obtained by multiplying a first artistic effectimage by a coefficient b changing over time with an image obtained bymultiplying a second artistic effect image different from the firstartistic effect image by a coefficient 1−b, and the coefficient b ischanged in response to a start trigger input of the transit processing.

The above and other features, advantages and technical and industrialsignificance of this invention will be better understood by reading thefollowing detailed description of presently preferred embodiments of theinvention, when considered in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of an imagingdevice according to a first embodiment of the present invention;

FIG. 2 is a perspective view illustrating a configuration of the imagingdevice according to the first embodiment of the present invention, whichis viewed at a side (front side) facing a user;

FIG. 3 is a diagram illustrating an outline of artistic effect imagingprocessing performed by an artistic effect image processing unit of theimaging device according to the first embodiment of the presentinvention;

FIG. 4 is a diagram for describing an outline of a multiecho;

FIG. 5 is a diagram for describing an outline of a one-shot echo;

FIG. 6 is a diagram illustrating a temporal change in a coefficient ofan SDRAM image to be synthesized with a most recently captured image;

FIG. 7 is a diagram for describing an outline of a transit;

FIG. 8 is a diagram illustrating a temporal change in a coefficient ofan artistic effect applied before a start trigger is input in transitprocessing;

FIG. 9 is a diagram illustrating an outline of a fluctuation;

FIG. 10 is a diagram illustrating an example of assigning an operationto each arrow key of a cross key at the time of moving image recordingand at the time of moving image non-recording;

FIG. 11 is a diagram illustrating an example (second example) ofassigning an operation to each arrow key of a cross key at the time ofmoving image recording and at the time of moving image non-recording;

FIG. 12 is a timing chart illustrating an example of synchronouscommunication between a lens control unit and a control unit of theimaging device according to the first embodiment of the presentinvention;

FIGS. 13A and 13B are flowcharts illustrating an outline of processingperformed by the imaging device according to the first embodiment of thepresent invention;

FIG. 14 is a diagram illustrating a screen display example for a movingimage in a display unit of the imaging device according to the firstembodiment of the present invention;

FIG. 15 is a flowchart illustrating an outline of moving image specialeffect processing performed by the imaging device according to the firstembodiment of the present invention;

FIG. 16 is a flowchart illustrating an outline of image processingperformed by the imaging device according to the first embodiment of thepresent invention;

FIG. 17 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 1-1 ofthe first embodiment of the present invention;

FIG. 18 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 1-2 ofthe first embodiment of the present invention;

FIG. 19 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 1-3 ofthe first embodiment of the present invention;

FIG. 20 is a flowchart illustrating an outline of moving image specialeffect processing performed by an imaging device according to ModifiedExample 1-4 of the first embodiment of the present invention;

FIG. 21 is a block diagram illustrating a configuration of an imagingdevice according to a second embodiment of the present invention;

FIG. 22 is a diagram illustrating an assignment of a user interfaceaccording to the second embodiment of the present invention;

FIG. 23 is a diagram illustrating an outline of a state transition of animaging device according to the second embodiment of the presentinvention at the time of moving image recording;

FIG. 24 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 2-1 ofthe second embodiment of the present invention;

FIG. 25 is a diagram illustrating a screen display example in a displayunit when an imaging device according to Modified Example 2-2 of thesecond embodiment of the present invention is in a still image capturingstandby state;

FIG. 26 is a diagram illustrating a screen display example in a displayunit when the imaging device according to Modified Example 2-2 of thesecond embodiment of the present invention is in a moving imagecapturing standby state;

FIG. 27 is a diagram illustrating a screen display example in a displayunit when a transit is selected as a moving image special effect;

FIG. 28 is a diagram illustrating a screen display example after amoving image button is pressed on a screen illustrated in FIG. 26;

FIG. 29 is a diagram illustrating a screen display example in a displayunit 21 after a decision icon is selected on a screen illustrated inFIG. 28;

FIG. 30 is a diagram illustrating a screen display example in a displayunit when a test on icon is selected on a screen illustrated in FIG. 26and so a test on state is set;

FIG. 31 is a block diagram illustrating a configuration of an imagingdevice according to a third embodiment of the present invention;

FIG. 32 is a diagram illustrating a key assignment when an imagingdevice according to the third embodiment of the present invention is setto a still image capturing mode or a moving image capturing mode;

FIGS. 33A and 33B are flowcharts illustrating an outline of processingperformed by the imaging device according to the third embodiment of thepresent invention;

FIG. 34 is a diagram illustrating a key assignment when an imagingdevice according to Modified Example 3-1 of the third embodiment of thepresent invention is set to a still image capturing mode or a movingimage capturing mode;

FIG. 35 is a diagram illustrating a screen display example in a displayunit when an imaging device according to Modified Example 3-2 of thethird embodiment of the present invention is in the still imagecapturing standby state;

FIG. 36 is a diagram illustrating a screen display example in a displayunit 21 when a shooting mode icon is selected on a screen illustrated inFIG. 35;

FIG. 37 is a diagram illustrating a screen display example in a displayunit in a moving image capturing standby state;

FIGS. 38A and 38B are flowcharts illustrating an outline of processingperformed by an imaging device according to a fourth embodiment of thepresent invention;

FIG. 39 is a block diagram illustrating a configuration of an imagingdevice according to a fifth embodiment of the present invention;

FIGS. 40A and 40B are flowcharts illustrating the details of imageprocessing performed by the imaging device according to the fifthembodiment of the present invention;

FIG. 41 is a diagram schematically illustrating an outline of resizingprocessing performed by a resizing processing unit of the imaging deviceaccording to the fifth embodiment of the present invention;

FIG. 42 is a diagram for schematically describing synthesis processingin a still image mode and a synthesis ratio c of a previous frame imagesignal;

FIG. 43 is a diagram schematically illustrating an outline of resizingprocessing performed by an imaging device according to Modified Example5-1 of the fifth embodiment of the present invention;

FIGS. 44A and 44B are flowcharts illustrating the details of imageprocessing performed by an imaging device according to an embodiment ofthe present invention;

FIG. 45 is a diagram schematically illustrating an outline of imageprocessing performed by an imaging device according to a seventhembodiment of the present invention;

FIGS. 46A and 46B are flowcharts illustrating an outline of imageprocessing performed by the imaging device according to the seventhembodiment of the present invention;

FIG. 47 is a diagram illustrating a relation between Bayer data and anexposure amount;

FIG. 48 is a diagram schematically illustrating an output line of a datastructure stored in an SDRAM of the imaging device according to theseventh embodiment of the present invention;

FIG. 49 is a flowchart illustrating an outline of multi-echo processing;

FIG. 50 is a diagram illustrating a relation between a value of an imagegain applied when an acquired image is a still image and a repeat count;

FIG. 51 is a diagram illustrating a relation between a value of an imagegain applied when an acquired image is a moving image and a repeatcount;

FIG. 52 is a diagram schematically illustrating a main part of a datastructure stored in an SDRAM; and

FIG. 53 is a diagram illustrating a form in which a ring buffer ismanaged using a queue.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments (hereinafter, referred to as an “embodiment”)for embodying the present invention will be described with reference tothe accompanying drawings. In the drawings, like reference numeralsdenote like parts.

First Embodiment

In a first embodiment of the present invention, when moving imagerecording starts using an imaging device having a still image capturingfunction and a moving image capturing function, a function of a userinterface for an operation input assigned for a still image is switchedto a function for a moving image special effect.

The imaging device of the first embodiment has a function of capturing astill image during moving image capturing. Here, examples of a method ofcapturing a still image during moving image capturing include a methodof capturing a still image in the process of capturing a moving imageand a method of capturing a still image directly after moving imagecapturing ends. Further, examples of the method of capturing a stillimage in the process of capturing a moving image includes a method ofstopping moving image capturing and then performing still imagecapturing and a method of performing still image capturing and movingimage capturing at the same time.

FIG. 1 is a block diagram illustrating a configuration of the imagingdevice of the first embodiment. FIG. 2 is a perspective viewillustrating a configuration of the imaging device according to thefirst embodiment of the present invention, which is viewed at a side(front side) facing the user. An imaging device 1 illustrated in FIGS. 1and 2 includes a main body unit 2 and a lens unit 3 detachably attachedto the main body unit 2.

The main body unit 2 includes a shutter 10, a shutter driving unit 11,an imaging element 12, an imaging element driving unit 13, a signalprocessing unit 14, an A/D (analog-to-digital) converting unit 15, animage processing unit 16, an AE (automatic exposure) processing unit 17,an AF (auto-focus) processing unit 18, an imagecompressing/decompressing unit 19, an input unit 20, a display unit 21,a display driving unit 22, a recording medium 23, a memory I/F 24, anSDRAM (Synchronous Dynamic Random Access Memory) 25, a flash memory 26,a main body communication unit 27, a bus 28, and a control unit 29.

The shutter 10 causes a state of the imaging element 12 to be set to anexposure state or a light-blocking state. The shutter driving unit 11 isconfigured using a stepping motor or the like, and drives the shutter 10in response to an instruction signal input from the control unit 29.

The imaging element 12 is configured, for example, using a CCD (ChargeCoupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) thatreceives light condensed by the lens unit 3 and converts the light intoan electric signal. The imaging element driving unit 13 causes theimaging element 12 to output image data (an analog signal) to the signalprocessing unit 14 at a predetermined timing. In this context, theimaging element driving unit 13 functions as an electronic shutter.

The signal processing unit 14 executes analog processing on the analogsignal input from the imaging element 12, and outputs a resultant signalto the A/D converting unit 15. Specifically, the signal processing unit14 performs noise reduction processing, gain-up processing, and the likeon the analog signal. For example, the signal processing unit 14 reducesreset noise and the like from the analog signal, performs waveformshaping, and then performs gain-up processing to cause brightness toreach to a target level.

The A/D converting unit 15 generates digital image data by performingA/D conversion on the analog signal input from the signal processingunit 14, and outputs the digital image data to the SDRAM 25 through thebus 28.

The image processing unit 16 acquires image data from the SDRAM 25through the bus 28, performs various kinds of image processing on theacquired image data (RAW data), and generates processed image data. Theprocessed image data is output to the SDRAM 25 through the bus 28. Theimage processing unit 16 includes a basic image processing unit 161, anartistic effect image processing unit 162, and a moving image specialeffect image processing unit 163.

The basic image processing unit 161 performs basic image processingincluding at least optical black subtraction processing, white balance(WB) adjustment processing, synchronization processing of image datawhen an imaging element has the Bayer array, color matrix calculationprocessing, gamma correction processing, color reproduction processing,and edge enhancement processing on image data. Further, the basic imageprocessing unit 161 performs finish effect processing of reproducing anatural image based on previously set parameters of respective imageprocessing, and so generates finish effect image data. Here, examples ofthe parameters of the respective image processing include a contrastvalue, a sharpness value, a chroma value, a white balance value, and agradation value.

The artistic effect image processing unit 162 performs artistic effectimage processing of causing a visual effect by combining a plurality ofimage processing on single image data, and so generates processed imagedata (hereinafter, referred to as “artistic effect image data”).

FIG. 3 is a diagram illustrating an outline of artistic effect imageprocessing performed by the artistic effect image processing unit 162.In FIG. 3, ten kinds of processing, that is, fantasic focus, fantasicfocus+starlight, fantasic focus+white edge, pop art, pop art+starlight,pop art+pinhole, pop art+white edge, toy photo, rough monochrome, anddiorama are described as artistic effect processing. Each artisticeffect image processing mentioned above will be described below.

The fantasic focus is processing of executing gradation processing on anentire image and giving an effect of a soft focus of synthesizing aresultant image with a non-gradated image at a predetermined ratio. Inthe fantasic focus, tone curve processing of causing intermediatebrightness to be brighter is performed, and thus an image of a beautifuland fantastic atmosphere appearing to be surrounded by happy light isformed or generated while leaving a detail of a subject in soft tone.For example, the fantasic focus is implemented by a combination of imageprocessing such as tone curve processing, gradation processing, alphablending processing, and image synthesis processing.

The fantasic focus+starlight is processing of applying a cross filtereffect of drawing a cross pattern on a high-brightness part of an imagein addition to the fantasic focus.

The fantasic focus+white edge is processing of applying an effect ofgradually taking on a white tint as it is closer from the center of animage to the edge thereof in addition to the fantasic focus. The whitetint effect is obtained by changing a pixel value such that the edgebecomes whiter as the distance from the center of an image increases.

The pop art is processing of enhancing a color to be colorful andrendering a bright and pleasant atmosphere. For example, the pop art isimplemented by a combination of chroma enhancement processing andcontrast enhancement processing. Overall, an effect of a high contrastand a high chroma is given.

The pop art+starlight is processing of applying the pop art and thestarlight in an overlapping manner. In this case, an effect in which across filter is applied to a colorful image is obtained.

The pop art+pinhole is processing of applying a toy photo (pinhole) thatdarkens the edge of an image by shading and gives an effect of lookingthrough a hole in addition to the pop art. The details of the toy photowill be described later.

The pop art+white edge is processing of applying the pop art and thewhite edge in an overlapping manner.

The toy photo is processing of causing brightness to decrease (darken)as the distance from the center of an image increases and giving aneffect of being sucked into unusual space when looking through a hole.For example, the toy photo is implemented by a combination of imageprocessing such as shading processing of multiplying a brightness signalby a coefficient whose value decreases as it is closer to a peripheralpart in addition to low pass filter processing, white balanceprocessing, contrast processing, and hue/chroma processing (for example,see Japanese Laid-open Patent Publication No. 2010-74244 for the detailsof the toy photo and the shading).

The rough monochrome is processing of adding a high contrast andgranular noise of a film and rendering a dynamic or rough monochromeimage. For example, the rough monochrome is implemented by a combinationof edge enhancement processing, level correction optimizationprocessing, noise pattern overlap processing, synthesis processing,contrast processing, and the like (for example, see Japanese Laid-openPatent Publication No. 2010-62836 for the details of the roughmonochrome). The noise pattern overlap processing (noise additionprocessing) is processing of adding a previously generated noise patternimage to an original image. For example, a random number may begenerated and used to generate the noise pattern image.

The diorama is processing of gradating the edge of an image of a highcontrast and a high chroma and so causing an atmosphere which felt as ifa miniature model or toy is being viewed to be created on a screen. Forexample, the diorama is implemented by a combination of hue/chromaprocessing, contrast processing, peripheral gradation processing,synthesis processing, and the like. Of these, in the peripheralgradation processing, low pass filter processing is performed whilechanging a low pass filter coefficient according to the position of animage such that a fading degree increases as the distance from thecenter of an image increases, that is, it is closer to a peripheralpart. Further, the peripheral gradation processing may be performed suchthat the top and bottom of an image or the left and right of an image isgradated.

The moving image special effect image processing unit 163 performs imageprocessing of adding a special effect to an image during moving imagerecording. Examples of a moving image special effect executed by themoving image special effect image processing unit 163 includes amultiecho, a one-shot echo, a transit, and a fluctuation. The movingimage special effects will be described below.

FIG. 4 is a diagram for describing an outline of a multiecho. Themultiecho refers to an effect of leaving an afterimage on an image byrepeatedly synthesizing an immediately previous recording image with acurrently captured image during a predetermined time period.Specifically, an effect by which the trajectory along which a ball 101moves is displayed as an after image is given as illustrated in FIG. 4.

Here, in a state in which the multiecho remains set as the moving imagespecial effect, when a start trigger is received, the moving imagespecial effect image processing unit 163 performs processing (multiechoprocessing) of synthesizing an image generated directly after that withan image of an immediately previous frame at a predetermined ratio.

In the example illustrated in FIG. 4, the moving image special effectimage processing unit 163 generates a synthesized image R1 bysynthesizing a captured image P1 captured directly after the starttrigger is input with a recording image R0 corresponding to a capturedimage P0 of an immediately previous frame. At the time of thissynthesis, in each pixel, a signal of the captured image P1 ismultiplied by a coefficient 0.6, and a signal of the recording image R0is multiplied by a coefficient 0.4.

Next, the moving image special effect image processing unit 163generates a multiecho image R2 by synthesizing the synthesized image R1with a captured image P2 of a next frame. Here, in order to generate themultiecho image R2, in each pixel, a signal of the captured image P2 ismultiplied by a coefficient 0.6, a signal of the synthesized image R1 ismultiplied by a coefficient 0.4.

The moving image special effect image processing unit 163 repeatedlyperforms the above-described synthesis processing until an end triggeris input and thus sequentially generates multiecho images. Here, whenthe end trigger is input, a recording image R5 corresponding to acaptured image P5 is recorded in the recording medium 23.

FIG. 5 is a diagram for describing an outline of a one-shot echo. Theone-shot echo refers to an effect by which an image (a momentary image)of a certain frame steadily disappears while becoming thinner.Specifically, an effect by which the ball 101 captured at a certain timeremains on a captured image as an afterimage for a while as illustratedin FIG. 5 is given.

Here, in a state in which the one-shot echo remains set as the movingimage special effect, when the start trigger is received, the movingimage special effect image processing unit 163 performs processing(one-shot echo processing) of storing image data, which has beencaptured directly after that and recorded, in the SDRAM 25 and thensynthesizing the image data with image data captured after that suchthat a weight steadily decreases over time.

In the example illustrated in FIG. 5, a captured image P1 captureddirectly after the start trigger is input is stored in the SDRAM 25(hereinafter, referred to as an “SDRAM image S1”). The moving imagespecial effect image processing unit 163 synthesizes the SDRAM image S1stored in the SDRAM 25 with a most recently captured image at apredetermined ratio.

In the example illustrated in FIG. 5, a one-shot echo image R11corresponding to the captured image P1 is displayed directly after thestart trigger is input.

Thereafter, when a time t is t₂, the moving image special effect imageprocessing unit 163 generates a one-shot echo image R12 by synthesizingthe SDRAM image S1 with a captured image P2. At the time of thissynthesis, in each pixel, a signal of the SDRAM image S1 is multipliedby a coefficient a (=0.8), and a signal of the captured image P2 ismultiplied by a coefficient 1−a (=0.2).

Thereafter, when the time t is t₃, the moving image special effect imageprocessing unit 163 generates a one-shot echo image R13 synthesized suchthat the coefficient a by which a signal of the SDRAM image S1 ismultiplied is set to 0.4, and the coefficient 1−a by which a signal of acaptured image P3 is multiplied is set to 0.6.

Thereafter, when the time t is t₄, the moving image special effect imageprocessing unit 163 generates a one-shot echo image R14 synthesized suchthat the coefficient a by which a signal of the SDRAM image S1 ismultiplied is set to 0.2 and the coefficient 1−a by which a signal of acaptured image P4 is multiplied is set to 0.8.

Thereafter, when the time t is t₅, the coefficient a becomes zero (0), acaptured image P5 is used as a recording image R5.

FIG. 6 is a diagram illustrating a temporal change in a synthesis ratioof an SDRAM image to be synthesized with a most recently captured image.In FIG. 6, a horizontal axis t represents a time elapsed from a point intime (t=0) at which the start trigger is input, and a vertical axis arepresents a coefficient by which each pixel signal of the SDRAM imageS1 is multiplied. As illustrated in FIG. 6, the coefficient a becomes a₁when the time t is zero (0), decreases smoothly over time, and thenbecomes zero (0) at a point in time at which a predetermined time t₅elapses after the start trigger is input. In other words, the ratio atwhich the SDRAM image S1 occupies in the synthesized image is set tosteadily decrease over time. Thus, an image appearing as if the SDRAMimage S1 steadily disappears over time can be generated.

The curved line illustrated in FIG. 6 is merely an example, and as faras a curved line smoothly changes such that the coefficient a decreasesover time and becomes zero (0) at a predetermined time, any curved linemay be applied. For example, a curved line including a time period forwhich the coefficient a has a predetermined value may be applied.

Further, the coefficient a may be decided according to the number offrames after the start trigger is input. In this case, it is preferablethat a value of the coefficient a steadily decrease in units of frames,and then the coefficient become zero (0) at a predetermined number offrames (for example, 120 frames).

FIG. 7 is a diagram for describing an outline of a transit. The transitrefers to an effect by which an artistic effect to be applied to oneimage changes to another artistic effect while steadily changing a ratiothereof. FIG. 7 schematically illustrates the transit that changes froman artistic effect A to an artistic effect B. In FIG. 7, directly beforea start trigger for the transit is input, an artistic effect image A0 inwhich the artistic effect A is applied to a captured image P0 isgenerated. A time taken from a switching start of an artistic effect bythe transit to a switching end may be arbitrarily set.

In the example illustrated in FIG. 7, the artistic effect A is assumedto be set before the start trigger is input. In other words, it isassumed that directly before the start trigger is input, the artisticeffect image processing unit 162 has generated an artistic effect imageR20 in which the artistic effect A is applied to the captured image P0.

Then, when the start trigger is input, the moving image special effectimage processing unit 163 generates a transit image R21 such that animage in which an artistic effect image A1 (a first artistic effectimage) obtained by applying the artistic effect A to a captured image P1is multiplied by a coefficient b (=b₀) of 0.8 is synthesized with animage in which an the artistic effect image B1 (a second artistic effectimage) obtained by applying the artistic effect B to the captured imageP1 is multiplied by a coefficient 1−b (=1−b₀) of 0.2.

Thereafter, when the time t is t₁₂, the moving image special effectimage processing unit 163 generates a transit image R22 synthesized suchthat the coefficient b by which a signal of an artistic effect image A2obtained by applying the artistic effect A to a captured image P2 ismultiplied is set to 0.6, and the coefficient 1-b by which a signal ofan artistic effect image B2 obtained by applying the artistic effect Bto the captured image P2 is multiplied is set to 0.4.

Thereafter, when the time t is t₁₃, the moving image special effectimage processing unit 163 generates a transit image R23 synthesized suchthat the coefficient b by which a signal of an artistic effect image A3obtained by applying the artistic effect A to a captured image P3 ismultiplied is set to 0.4, and the coefficient 1−b by which a signal ofan artistic effect image B3 obtained by applying the artistic effect Bto the captured image P3 is multiplied is set to 0.6.

Thereafter, when the time t is t₁₄, the moving image special effectimage processing unit 163 generates a transit image R24 synthesized suchthat the coefficient b by which a signal of an artistic effect image A4obtained by applying the artistic effect A to a captured image P4 ismultiplied is set to 0.2, and the coefficient 1−b by which a signal ofan artistic effect image B4 obtained by applying the artistic effect Bto the captured image P4 is multiplied is set to 0.8.

Thereafter, when the time t is t₁₅, the moving image special effectimage processing unit 163 generates a transit image R25 in which theartistic effect B is applied to the captured image P5 (b=0).

FIG. 8 is a diagram illustrating a temporal change in the synthesisratio (coefficient) b of the artistic effect applied before the starttrigger is input in the transit processing. As illustrated in FIG. 8,the coefficient b is b₀ when the time t is zero (0), smoothly decreasesover time, and then becomes zero (0) at a time in point when apredetermined time t₁₅ elapses after the start trigger is input.

The curved line illustrated in FIG. 8 are merely examples, and as far asa curved line smoothly changes such that the coefficient b decreasesover time and becomes zero (0) at a predetermined time, any curved linemay be applied. For example, a curved line including a time period forwhich the coefficient b has a predetermined value may be applied.

Further, the coefficient b may be decided according to the number offrames after the start trigger is input. In this case, it is preferablethat a value of the coefficient b smoothly decrease in units of frames,and the coefficient become zero (0) at a predetermined number of frames(for example, 120 frames).

In the first embodiment, during transit processing, an image obtained byapplying an artistic effect set directly before a start trigger is inputto a captured image is used as the first artistic effect image. However,an image obtained by applying an artistic effect set directly after astart trigger is input to a captured image may be used as the firstartistic effect image, or an image obtained by applying an artisticeffect set at the same time as when the start trigger is input to acaptured image may be used as the first artistic effect image.

FIG. 9 is a diagram illustrating an outline of a fluctuation. Thefluctuation refers to an effect obtained by randomly changing anapplication degree of an artistic effect to be applied to each frame ofa moving image over time. FIG. 9 illustrates an example in which anartistic effect is a cross filter. In this case, the moving imagespecial effect image processing unit 163 applies a fluctuation byrandomly rotating a cross 102 over time, and randomly changing the sizeof the cross 102 over time.

Continuously, the configuration of the imaging device 1 will bedescribed.

The AE processing unit 17 acquires image data recorded the SDRAM 25through the bus 28, and sets the exposure condition used when capturinga still image or a moving image based on the acquired image data.Specifically, the AE processing unit 17 performs automatic exposure ofthe imaging device 1 by calculating brightness based on image data andthen deciding a setting value of an aperture value (F value), a shutterspeed, and the like based on the calculated brightness.

The AF processing unit 18 acquires image data recorded in the SDRAM 25through the bus 28, and performs an automatic focal adjustment of theimaging device 1 based on the acquired image data. For example, the AFprocessing unit 18 performs an automatic focal adjustment of the imagingdevice 1 such that focus evaluation of the imaging device 1 is decidedby extracting a signal of a high frequency component from image data andperforming an AF (Auto Focus) calculation process on the signal of thehigh frequency component.

The image compressing/decompressing unit 19 acquires image data recordedin the SDRAM 25 through the bus 28, compresses the acquired imageaccording to a predetermined format, and then outputs the compressedimage data to the SDRAM 25. Here, the JPEG (Joint Photographic ExpertsGroup) format, the Motion JPEG format, the MP4 (H.264), or the like maybe used as the predetermined format. Further, the imagecompressing/decompressing unit 19 acquires image data (compressed imagedata) recorded in the recording medium 23 through the bus 28 and thememory I/F 24, decompresses the acquired image data, and then outputsthe decompressed image data to the SDRAM 25. Instead of the recordingmedium 23, a storage unit may be disposed in the imaging device 1.

The input unit 20 includes a setting signal input unit 201 that receivesan input of various kinds of setting signals including a still imagecapturing condition setting signal used to set a capturing condition onstill image capturing and moving image capturing, and a moving imagecapturing signal input unit 202 that receives an input of a moving imagecapturing start signal instructing at least moving image capturing tostart.

The input unit 20 is implemented using a user interface for an operationsignal input disposed on the surface of the main body unit 2. Next, aconfiguration of the user interface serving as a part of the input unit20 will be described.

As the user interface for the operation signal input, the imaging device1 includes a power button 41 to switch a power state of the imagingdevice 1 to an on state or an off state, a release button 42 to receivean input of a still image release signal used to give a still imagecapturing instruction, a mode dial 43 to change various kinds ofshooting modes set to the imaging device 1, an operation button 44 tochange various kinds of settings of the imaging device 1, a menu button45 to cause various kinds of settings of the imaging device 1 to bedisplayed on the display unit 21, a playback button 46 to cause an imagecorresponding to image data recorded in the recording medium 23 to bedisplayed on the display unit 21, a moving image button 47 to receive aninput of a moving image release signal giving a moving image capturinginstruction, a function key 48 to set various kinds of functions of theimaging device 1, an erase button 49 to erase data, and a touch panel 50disposed to be superimposed on a display screen of the display unit 21and used to receive an input signal corresponding to a contact positionfrom the outside.

The release button 42 moves forward or backward by external pressure.Here, when the release button 42 is pressed halfway, a first releasesignal instructing a shooting preparation operation is input. On theother hand, when the release button 42 is fully pressed, a secondrelease signal instructing still image capturing is input.

The operation button 44 includes a cross key 441 forming a cross shapein top, bottom, left, and right directions to perform a selectionsetting input in a menu screen or the like, and a decision button 442 todecide selection by the cross key 441. The cross key 441 includes anup-arrow key 443, a down-arrow key 444, a left-arrow key 445, and aright-arrow key 446.

In the user interface described above, the buttons other than the movingimage button 47 configure a part of the setting signal input unit 201.The moving image button 47 configures a part of the moving imagecapturing signal input unit 202.

FIG. 10 is a diagram illustrating an example of assigning an operationto each arrow of the cross key 441 when it is not in the process ofmoving image recording (during moving image non-recording) and when itis in the process of moving image recording (during moving imagerecording). Here, when it is not in the process of moving imagerecording, an exposure correction operation is assigned to the up-arrowkey 443. A continuous shooting/single shooting operation is assigned tothe down-arrow key 444. An AF target operation is assigned to theleft-arrow key 445. A flash setting operation is assigned to theright-arrow key 446.

Next, an example when it is in the process of moving image recordingwill be described. In this case, a one-shot echo start operation isassigned to the up-arrow key 443. A transit start operation is assignedto the down-arrow key 444. The left-arrow key 445 functions as a togglekey to alternately assign a multiecho start operation and a multiechoend operation. A fluctuation start operation is assigned to theright-arrow key 446. Further, when a setting is made such that the endof the fluctuation can be arbitrarily selected, it is desirable to causethe right-arrow key 446 to function as a toggle key to alternatelyoperate a start operation and an end operation of the fluctuation.

An assignment of an input button at the time of moving image recordingis not limited to the example illustrated in FIG. 10. For example, asillustrated in FIG. 11, an effect start operation may be assigned to theup-arrow key 443, an effect stop operation may be assigned to thedown-arrow key 444, and an effect switching operation may be assigned tothe left-arrow key 445 and the right-arrow key 446. In this case, setspecial effects are preferably displayed on the display unit 21.

Further, an operation assignment of the cross key 441 may be setaccording to usability or the frequency of use of an operation or may beuniquely set by the user. Further, there may be made a setting thatcauses another operation to be performed when both of neighboring twokeys (for example, the up-arrow key 443 and the right-arrow key 446) arepressed.

The display unit 21 is configured using a liquid crystal display panel,an organic EL (Electro Luminescence) display panel, or the like. Thedisplay driving unit 22 acquires image data stored in the SDRAM 25 orimage data stored in the recording medium 23 through the bus 28, andcauses an image corresponding to the acquired image data to be displayedon the display unit 21. Here, examples of a display of an image includea rec-view display in which an image data directly after shooting isdisplayed only for a predetermined time (for example, three seconds), aplayback display in which image data stored in the recording medium 23is played back, and a live-view display in which live-view imagescorresponding to image data continuously generated by the imagingelement 12 are sequentially displayed in time series. Further, thedisplay unit 21 appropriately displays operation information andshooting-related information of the imaging device 1.

The recording medium 23 is configured using, for example, a memory cardmounted from the outside of the imaging device 1. The recording medium23 is removably mounted to the imaging device 1 through the memory I/F24. Image data processed by the image processing unit 16 or the imagecompressing/decompressing unit 19 is written in the recording medium 23through a reading/writing device (not illustrated) corresponding to akind thereof, or image data recorded in the recording medium 23 is readthrough the reading/writing device. In addition, the recording medium 23may output imaging program and various kinds of information to the flashmemory 26 through the memory I/F 24 and the bus 28 under control of thecontrol unit 29.

The SDRAM 25 is configured using a volatile memory. The SDRAM 25 has afunction as a primary storage unit that temporarily stores image datainput from the A/D converting unit 15 through the bus 28, processedimage data input from the image processing unit 16, and informationwhich is being processed by the imaging device 1. For example, the SDRAM25 temporarily stores image data sequentially output in units of framesfrom the imaging element 12 through the signal processing unit 14, theA/D converting unit 15, and the bus 28.

The flash memory 26 is configured using a non-volatile memory. The flashmemory 26 includes a program recording unit 261, a special effectprocessing information recording unit 262, and an image processinginformation recording unit 263. The program recording unit 261 recordsvarious kinds of programs for operating the imaging device 1, an imagingprogram, various kinds of data used during execution of a program,various kinds of parameters necessary for an image processing operationby the image processing unit 16, and the like. The special effectprocessing information recording unit 262 records image processingcombination information in each artistic effect image processingperformed by the artistic effect image processing unit 162. The imageprocessing information recording unit 263 records image processinginformation in which a processing time is associated with imageprocessing executable by the image processing unit 16. In addition, theflash memory 26 records a manufacturing number specifying the imagingdevice 1 and the like.

The main body communication unit 27 is a communication interface forperforming communication with the lens unit 3 mounted to the main bodyunit 2. The main body communication unit 27 also includes an electricalcontact point with the lens unit 3.

The bus 28 is configured using, for example, a transmission pathconnecting the respective components of the imaging device 1. The bus 28transfers various kinds of data internally generated in the imagingdevice 1 to the respective components of the imaging device 1.

The control unit 29 is configured using a CPU (Central Processing Unit)or the like. The control unit 29 includes an image processing controlunit 291 and a display control unit 292.

The image processing control unit 291 sets content of image processingto be executed by the image processing unit 16 in response to aninstruction signal from the input unit 20, which is input through thebus 28, and causes any one of the basic image processing unit 161, theartistic effect image processing unit 162, and the moving image specialeffect image processing unit 163 to execute image processing accordingto the set content.

The display control unit 292 controls a display form of the display unit21. Specifically, the display control unit 292 drives the displaydriving unit 22, and causes an image corresponding to various kinds ofimage data processed by the image processing unit 16 to be displayed onthe display unit 21.

The control unit 29 controls an operation of the imaging device 1 ingeneral by transmitting a control signal or various kinds of data to therespective components configuring the imaging device 1 in response to aninstruction signal transmitted from the input unit 20 through the bus28.

Here, when the second release signal is input through the release button42, the control unit 29 performs control such that the imaging device 1starts a still image capturing operation. Further, when the moving imagecapturing start signal is input through the moving image button 47, thecontrol unit 29 performs control such that the imaging device 1 starts amoving image capturing operation. Here, the image capturing operation inthe imaging device 1 refers to an operation in which the signalprocessing unit 14, the A/D converting unit 15, and the image processingunit 16 execute predetermined processing on image data which the imagingelement 12 has output according to driving of the shutter driving unit11 and the imaging element driving unit 13. The image data processed inthe above-described way is compressed according to a predeterminedformat through the image compressing/decompressing unit 19, and thenrecorded in the recording medium 23 through the bus 28 and the memoryI/F 24 under control of the image processing control unit 291. In thefirst embodiment, the recording medium 23 serves a part of the storageunit. However, separately from the recording medium 23, a storage areahaving a function of a storage unit is secured inside the imaging device1, and compressed image data may be stored in the storage area.

The main body unit 2 having the above-described configuration mayfurther include an audio input/output unit, a fill light emitting unitthat emits a fill light (flash) to a subject, a communication unithaving a function of performing two-way communication with an externaldevice through the Internet, and the like.

Next, a configuration of the lens unit 3 will be described. The lensunit 3 includes an optical system 31, a lens driving unit 32, adiaphragm 33, a diaphragm driving unit 34, a lens operating unit 35, alens flash memory 36, a lens communication unit 37, and a lens controlunit 38.

The optical system 31 is configured using one or more lenses. Theoptical system 31 condenses light from a predetermined field region. Theoptical system 31 has an optical zoom function of changing an angle ofview and a focus function of changing a focus.

The lens driving unit 32 is configured using a direct current (DC)motor, a stepping motor, or the like, and changes, for example, a focusposition or an angle of view of the optical system 31 by moving a lensof the optical system 31 on an optical axis L.

The diaphragm 33 adjusts exposure by limiting an amount of incidentlight condensed by the optical system 31.

The diaphragm driving unit 34 is configured using a stepping motor orthe like, and drives the diaphragm 33.

The lens operating unit 35 is a ring disposed around a lens barrel ofthe lens unit 3 as illustrated in FIG. 2, and receives an input of anoperation signal to start an optical zoom operation in the lens unit 3or an input of an instruction signal instructing a focus positionadjustment in the lens unit 3. The lens operating unit 35 may be apush-type switch or the like.

The lens flash memory 36 records a control program that decides theposition and movement of the optical system 31, lens characteristics ofthe optical system 31, and various kinds of parameters.

The lens communication unit 37 is a communication interface thatperforms communication with the main body communication unit 27 of themain body unit 2 when the lens unit 3 is mounted to the main body unit2. The lens communication unit 37 also includes an electrical contactpoint with the main body unit 2.

The lens control unit 38 is configured using a CPU (Central ProcessingUnit) or the like. The lens control unit 38 controls an operation of thelens unit 3 in response to an operation signal of the lens operatingunit 35 or an instruction signal from the main body unit 2.Specifically, in response to the operation signal of the lens operatingunit 35, the lens control unit 38 drives the lens driving unit 32 toperform a focusing operation or a zoom change operation of the lens unit3, and drives the diaphragm driving unit 34 to change an aperture value.Further, the lens control unit 38 may be configured to transmit focusposition information of the lens unit 3, a focus distance, specificinformation identifying the lens unit 3, and the like to the main bodyunit 2 when the lens unit 3 is mounted to the main body unit 2.

The lens control unit 38 promotes operation cooperation with the mainbody unit 2 by exchanging a lens communication signal with the controlunit 29 of the main body unit 2 at a predetermined period. FIG. 12 is atiming chart illustrating an example of synchronous communicationbetween the lens control unit 38 and the control unit 29. FIG. 12( a)illustrates processing performed inside the main body unit 2. FIG. 12(b) illustrates a vertical synchronous signal. FIG. 12( c) illustrates animaging timing and a reading timing. FIG. 12( d) illustrates lenscommunication. FIG. 12( e) illustrates a lens communication synchronoussignal. FIG. 12( f) illustrates a lens position acquisition signal. FIG.12( g) illustrates processing performed inside the lens unit 3.

First, the control unit 29 causes the image processing unit 16 toexecute image processing and a calculation of an AF evaluation value oflive-view image based on image data acquired in a previous frame, andtransmits a lens state data request command used to acquire lens statedata to the lens control unit 38 (B1 and BL). At this time, the controlunit 29 transmits a synchronous signal for lens communication and a lensposition acquisition signal instructing a timing to acquire positioninformation of the optical system 31 at the same period as the verticalsynchronous signal in a synchronous communication mode. The lensposition acquisition signal is a signal whose state changes at a pointin time at which half an accumulation time of a central portion of theimaging element 12 elapses as illustrated in FIG. 12( c).

The lens control unit 38 acquires position information of the opticalsystem 31 at a timing at which the state of the lens positionacquisition signal changes, and detects an operation state of the lensoperating unit 35 at a reception timing of the lens communicationsynchronous signal (L1).

Next, the lens control unit 38 transmits lens state data including theposition information of the optical system 31 and the detection state ofthe lens operating unit 35 which are acquired in the process L1 to thecontrol unit 29 in response to the lens state data request commandreceived from the control unit 29 (L2).

Thereafter, the control unit 29 performs a calculation of the AFevaluation value and various kinds of setting changes such as anexposure value change based on the lens state data transmitted from thelens control unit 38 (B2).

The control unit 29 and the lens control unit 38 repeatedly perform theabove-described processing at regular intervals.

FIGS. 13A and 13B are flowcharts illustrating an outline of processingperformed by the imaging device 1 including the above-describedconfiguration. Referring to FIGS. 13A and 13B, first, when the useroperates the power button 41 to turn power of the imaging device 1 on(Yes in step S1), the control unit 29 initializes the imaging device 1(step S2). In this initialization processing, for example, the controlunit 29 performs processing of resetting a recording flag representingthat a moving image is being recorded to an off state, resetting aspecial effect flag representing the application related to a specialeffect in a moving image to an off state, and starting lenscommunication (see FIG. 12) with the lens control unit 38. However, whenpower of the imaging device 1 is not turned on (No in step S1), theimaging device 1 repeats step S1.

Next, when the playback button 46 is not operated (No in step S3) andthe menu button 45 is operated (Yes in step S4), the imaging device 1displays a display screen used to change a setting, and executes settingprocessing of setting various kinds of conditions of the imaging device1 in response to the user's selection operation (step S5). After stepS5, the imaging device 1 causes the process to proceed to step S7 whichwill be described later.

Here, examples of content to be set includes finish effect processing,artistic effect image processing, a still image recording mode, a movingimage recording mode, and moving image special effect processing. Forexample, the finish effect processing includes natural processing whichis processing of finishing an image in a natural tone, vivid processingwhich is processing of finishing an image vividly, flat processing whichis processing of finishing with the emphasis on a material property of asubject, and monotone processing which is processing of finishing animage in a monochrome tone. Examples of the still image recording modeincludes a JPEG recording mode, a JPEG+RAW recording mode, and a RAWrecording mode depending on the type of still image of a recordingtarget. The moving image capturing mode is a mode decided according to amoving image compression format, and includes, for example, aMotion-JPEG mode and an MP4 (H.264) mode. For example, a switchingdestination of an artistic effect by transit processing, a fluctuation,and the like are set through the moving image special effect processing.

Here, when the playback button 46 is operated in step S3 (Yes in stepS3), the imaging device 1 performs playback processing (step S6). Instep S6, the display control unit 292 causes a list of files recorded inthe recording medium 23 to be displayed on the display unit 21.Thereafter, when a playback image is selected and input through theinput unit 20, image data is acquired from the recording medium 23, theacquired image data is decompressed through the imagecompressing/decompressing unit 19, and then displayed on the displayunit 21. Thereafter, the imaging device 1 causes the process to proceedto step S18.

Here, when the playback button 46 is not operated in step S3 (No in stepS3), the menu button 45 is not operated (No in step S4), and the movingimage button 47 is operated (Yes in step S7), the control unit 29inverts the recording flag representing that a moving image is beingrecorded (step S8). Specifically, for example, when the recording flagis in the on state, the control unit 29 sets the recording flag to theoff state.

Next, the control unit 29 determines whether or not the recording flagrecorded in the SDRAM 25 is in the on state (step S9). Here, when it isdetermined that the recording flag is in the on state (Yes in step S9),the control unit 29 generates a moving image file used to record imagedata in the recording medium 23 in time series and stores the movingimage file in the recording medium 23 (step S10).

Thereafter, the control unit 29 sets a user interface (UI) for a specialmoving image (step S11). Through this setting, for example, anassignment at the time of moving image recording illustrated in FIG. 10is executed. The control unit 29 recognizes a signal subsequentlyreceived by the user interface which is the setting target in step S11as a signal for a special moving image based on FIG. 10.

Next, the display control unit 292 changes, for example, a setting of anon-screen display (OSD) to be displayed on the display unit 21 andperforms switching to a screen for a moving image (step S12).Specifically, for example, the display control unit 292 displays aremaining time, an icon representing that a special effect can beapplied during moving image capturing, and the like. Thereafter, theimaging device 1 causes the process to proceed to step S15 which will bedescribed later.

FIG. 14 is a diagram illustrating a screen display example for a movingimage in the display unit 21. As illustrated in FIG. 14, a moving imageicon 111 representing that a moving image special effect can be applied,a on screen display 112 representing a screen key assignment for amoving image, and a remaining time 113 displaying a remaining time ofmoving image capturing are displayed on a screen Q for a moving image.Of these, the on screen display 112 may be displayed for several secondsand then automatically disappeared or may be constantly displayed.Further, the on screen display 112 may be displayed or hidden accordingto an input from the setting signal input unit 201.

Here, when it is determined in step S9 that the recording flag is in theoff state (No in step S9), the control unit 29 sets a user interface fora still image (step S13).

Next, the display control unit 292 switches a setting of the on screendisplay in the display unit 21 to a setting for a still image (stepS14). Through this switching, for example, the display unit 21 displaysthe number of remaining records, an icon representing that a specialeffect can be applied during still image capturing, and the like.Thereafter, the imaging device 1 causes the process to proceed to stepS15 which will be described later.

Meanwhile, when the moving image button 47 is not operated in step S7(No in step S7), the image processing control unit 291 causes the movingimage special effect image processing unit 163 to perform moving imagespecial effect processing (step S15).

FIG. 15 is a flowchart illustrating an outline of moving image specialeffect processing. Referring to FIG. 15, when a key used to start amoving image special effect is operated (Yes in step S31), the imageprocessing control unit 291 sets a moving image special effect flag toan on state (step S32). Thereafter, the image processing control unit291 performs effect setting (step S33).

However, when a key used to start a moving image special effect is notoperated (No in step S31) and then an end instruction is input (Yes instep S34), the image processing control unit 291 performs control suchthat the moving image special effect flag is set to an off state (stepS35). Thereafter, the imaging device 1 returns to the main routine.

On the other hand, when a key used to start a moving image specialeffect is not operated (No in step S31) and then an end instruction isnot input (No in step S34), the imaging device 1 returns to the mainroutine.

After moving image special effect processing of step S15, when the firstrelease signal is input from the release button 42 (Yes in step S16),the control unit 29 causes the AE processing unit 17 to execute AEprocessing of adjusting exposure, and causes the AF processing unit 18to execute AF processing of adjusting a focus (step S17).

Next, the control unit 29 determines whether or not power of the imagingdevice 1 has been turned off by an operation on the power button 41(step S18). Here, when the control unit 29 determines that power of theimaging device 1 has been turned off (Yes in step S18), the imagingdevice 1 ends the current process. However, when the control unit 29determines that power of the imaging device 1 has not been turned off(No in step S18), the imaging device 1 causes the process to return tostep S3.

Meanwhile, when the first release signal is not input from the releasebutton 42 (No in step S16) and the second release signal is input fromthe release button 42 (Yes in step S19), the control unit 29 drives theshutter driving unit 11 and the imaging element driving unit 13 andperforms shooting by a mechanical shutter (step S20).

Next, the image processing unit 16 executes predetermined imageprocessing on a captured still image (step S21). The details of imageprocessing will be described later.

Thereafter, the control unit 29 compresses image data in a JPEG formatthrough the image compressing/decompressing unit 19, and records thecompressed image data in the recording medium 23 (step S22). In stepS22, the control unit 29 may record the image data compressed in theJPEG format through the image compressing/decompressing unit 19 in therecording medium 23 in association with RAW data which has not beensubjected to image processing by the image processing unit 16. Afterstep S22, the imaging device 1 causes the process to proceed to stepS18.

Meanwhile, when the second release signal is not input from the releasebutton 42 in step S19 (step S19: No), the control unit 29 causes the AEprocessing unit 17 to execute AE processing of adjusting exposure, andcauses the AF processing unit 18 to execute AF processing of adjusting afocus (step S23).

Next, the control unit 29 drives the imaging element driving unit 13 andperforms shooting by an electronic shutter (step S24).

Thereafter, the image processing unit 16 performs image processing basedon setting information of the imaging device 1 (step S25). The detailsof image processing will be described later.

The display control unit 292 causes live-view image corresponding to theimage data processed by the image processing unit 16 to be displayed onthe display unit 21 (step S26).

Next, when the imaging device 1 is in the process of moving imagerecording (Yes in step S27), the control unit 29 compresses image datathrough the image compressing/decompressing unit 19, and records thecompressed image data in a moving image file created in the recordingmedium 23 as a moving image (step S28). Thereafter, the imaging device 1causes the process to proceed to step S18. However, when the imagingdevice 1 is not in the process of moving image recording in step S27 (Noin step S27), the imaging device 1 causes the process to proceed to stepS18.

FIG. 16 is a flowchart illustrating an outline of image processing.Referring to FIG. 16, the basic image processing unit 161 performs basicimage processing (step S41). Here, the basic image processing includesprocessing such as subtraction of an OB (Optical Black) value, WBcorrection, synchronization, a color matrix calculation, gammaconversion color correction, edge enhancement, and NR (Noise Reduction).

Here, the WB correction is processing of perform correction bymultiplying image data of the Bayer array by an R gain and a B gaincorresponding to a WB mode previously set by the user, reads the WB froma flash memory from the imaging device main body, and multiplying by thevalue.

The synchronization is processing of interpolating data not included ina corresponding pixel from the periphery and converting the data intodata in which each pixel is configured with RGB data when the imagingelement 12 has the Bayer array.

The color matrix calculation is processing of reading a color matrixcoefficient corresponding to a set WB mode from the flash memory fromthe main body and multiplying the color matrix coefficient.

In the gamma conversion color correction processing, a gamma tablepreviously designed according to a finish setting is read from the flashmemory of the main body, and image data is subjected to gammaconversion. At this time, gamma conversion applied to RGB data may beperformed such that an RGB color space is converted into a color spacerepresented by a brightness signal Y and two color difference signals Cband Cr, and then gamma conversion is performed only on the brightnesssignal Y. In addition, in order to obtain appropriate colorreproducibility, color correction may be performed using a sideparameter previously designated according to a finish setting. A gammacurved line may be changed according to the type of artistic effect.

In edge enhancement processing, enhancement is performed such that anedge component is extracted by a band-pass filter, multiplied by acoefficient corresponding to an edge enhancement level, and added toimage data.

In NR processing, processing of reducing noise is performed such thatfrequency decomposition is performed on an image, and then coringprocessing is performed according to a frequency.

Next, the artistic effect image processing unit 162 performs artisticeffect image processing (step S42). Here, processing such as crossfilter, soft focus, noise addition, shading, peripheral brightnessincrease, and peripheral gradation are performed.

Thereafter, when the imaging device 1 performs still image capturing(Yes in step S43), the control unit 29 performs control such that aspecial effect (a second special effect) corresponding to a moving imagespecial effect (a first special effect) is inhibited from being appliedto image data of a still image, and causes the process to return to themain routine.

Here, the reason why a special effect corresponding to a moving imagespecial effect is not applied when the imaging device 1 performs stillimage capturing will be described. In still image capturing duringmoving image capturing, a response speed is important. Since the numberof pixels of a still image is several times larger than the number ofpixels of a moving image, when a moving image special effect is appliedto a still image, a restart timing of moving image capturing after stillimage capturing may be affected. In this regard, in the firstembodiment, a moving image special effect is inhibited from beingapplied to a still image. Further, when a response is not affected bylight flux, it can be applied to a still image.

Meanwhile, when the imaging device 1 performs moving image capturing (Noin step S43), the control unit 29 determines whether or not the movingimage special effect remains set (step S44). Here, when the moving imagespecial effect remains set (Yes in step S44), the imaging device 1causes the process to proceed to step S45. However, when the movingimage special effect does not remain set (No in step S44), the imagingdevice 1 returns to the main routine.

Here, in a state in which it is determined in step S44 that the movingimage special effect remains set (Yes in step S44), when an effect toapply is the multiecho (Yes in step S45), the moving image specialeffect image processing unit 163 performs multiecho processing (see FIG.4) such that an immediately previous image processing result (previousframe) is synthesized with a current frame at a predetermined ratio(step S46). However, when a special effect to apply is not the multiecho(No in step S45), the imaging device 1 causes the process to proceed tostep S47.

In step S47, the control unit 29 determines whether or not an effect toapply is the one-shot echo. Here, when an effect to apply is theone-shot echo (Yes in step S47), the moving image special effect imageprocessing unit 163 performs synthesis processing on a release framewhich is specific image data stored in the SDRAM 25 in order to obtain aone-shot echo effect (step S48). Here, in case of a first frame afterthe special effect flag is set to the on state, the moving image specialeffect image processing unit 163 performs processing of storing an imageof a current frame in the SDRAM 25. However, in case of a second orlater frame after the special effect flag is set to the on state, themoving image special effect image processing unit 163 performs synthesisprocessing of synthesizing a corresponding frame with the release framestored in the SDRAM 25.

Meanwhile, when it is determined in step S47 that a moving image specialeffect to apply is not a one-shot echo (No in step S47), the imagingdevice 1 causes the process to proceed to step S49.

In step S49, the control unit 29 determines whether or not an effect toapply is the fluctuation. Here, when an effect to apply is thefluctuation (Yes in step S49), the moving image special effect imageprocessing unit 163 performs processing of adding a fluctuation effect(step S50).

Here, concrete processing of the moving image special effect imageprocessing unit 163 in step S50 will be described. The moving imagespecial effect image processing unit 163 adds an effect of applyingfluctuation to an image processing parameter in artistic effect imageprocessing such as shading processing and cross filter processing. Forexample, in case of shading processing, an attenuation characteristicfrom the center of an image is changed over time. In case of peripheralgradation, a gradation amount or a gradation shape is changed. In caseof cross filter, the length or angle of a cross pattern is changed overtime. Further, chroma, contrast, or white balance may be changed overtime. Further, when the fluctuation overlaps an artistic effect suchshading, both may be applied, or only fluctuation may be applied.

Meanwhile, when it is determined in step S49 that an effect to apply isnot the fluctuation (No in step S49), the imaging device 1 causes theprocess to proceed to step S51.

In step S51, the control unit 29 determines whether or not an effect toapply is the transit (step S51). Here, when an effect to apply is thetransit (Yes in step S51), the basic image processing unit 161 and theartistic effect image processing unit 162 execute basic image processingand artistic effect image processing according to a finish/switchingdestination setting, respectively, (steps S52 and S53). Thereafter, themoving image special effect image processing unit 163 performsprocessing of synthesizing two images for transit processing and sogenerates a transit image (step S54). After step S54, the imaging device1 returns to the main routine.

Meanwhile, when the control unit 29 determines in step S51 that aneffect to apply is not the transit (No in step S51), the imaging device1 returns to the main routine.

According to the first embodiment described above, when moving imagerecording starts, the setting signal input unit is switched for a movingimage special effect, and the control unit recognizes that a settingsignal of an moving image special effect has been input. Thus, the usercan easily apply a special effect to a moving image by performing asimple operation at the time of shooting.

Further, according to the first embodiment, the user interfaceconfiguring a setting signal input unit can be provided with both aninput function for a moving image and an input function for a stillimage. Thus, the user interface having excellent operability even duringmoving image capturing can be implemented. In addition, since the numberof user interfaces can be suppressed, a limitation to the layout can bereduced, and an imaging device suitable for miniaturization can beimplemented.

Further, according to the first embodiment, a special effect equivalentto an effect which can be implemented on editing devices or personalcomputers (PCs) can be implemented by an imaging device. Thus, a movingimage in which the user's shooting intention is reflected can begenerated without requiring a high level of expertise and performingpost editing.

Further, according to the first embodiment, control is performed suchthat a moving image special effect is inhibited from being applied to astill image during moving image capturing, and thus influence on arestart timing of moving image capturing after still image capturing canbe prevented.

Generally, imaging devices are smaller in memory capacity and processingcapability than personal computers (PCs), and thus it is difficult toimplement moving image data editing processing equivalently of PCs orthe like. Further, in an imaging device capable of performing stillimage capturing during moving image capturing or performing still imagecapturing directly after moving image capturing ends, a memory shortageor a memory management problem is decisive, and there may occur aproblem in that the cost increase due to memory addition, responsivenessduring still image capturing is lowered, and a still image qualitydeteriorates. In this regard, in the first embodiment, implemented is atechnique capable of capturing a moving image in which a photographer'sshooting intension is reflected through a special effect of a temporalchange over a plurality of frames at the time of shooting withoutediting a captured moving image.

Modified Example 1-1

FIG. 17 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 1-1 ofthe first embodiment. In the example illustrated in FIG. 17, at the timeof moving image non-recording, the AE/AF operation is assigned to thefirst release operation of the release button 42, the shooting operationis assigned to the second release operation of the release button 42,and the focusing operation is assigned to the lens operating unit 35. Onthe other hand, at the time of moving image recording, the releasebutton 42 (the first release operation) functions as a toggle key toalternately operate an effect start operation and an effect stopoperation, and an effect switching operation is assigned to the lensoperating unit 35. Here, when the AF operation or the AE operationcorresponding to an input from the release button 42 is performed duringmoving image recording, or when a still image is captured when an effectstarts, a trouble may occur. In this regard, at the time of moving imagerecording, still image capturing is not performed.

Modified Example 1-2

FIG. 18 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 1-2 ofthe first embodiment. In this case, at the time of moving imagenon-recording, the AF operation is assigned to the function key 48, andthe focusing operation is assigned to the lens operating unit 35. On theother hand, at the time of moving image recording, the function key 48functions as a toggle key to alternately operate an effect startoperation and an effect stop operation, and the effect switchingoperation is assigned to the lens operating unit 35.

Modified Example 1-3

FIG. 19 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 1-3 ofthe first embodiment. In Modified Example 1-3, on/off switching of aneffect is performed using the touch panel 50. At the time of movingimage non-recording, the touch operation on the touch panel 50 isassigned to the still image capturing operation. On the other hand, atthe time of moving image recording, the touch operation on the touchpanel 50 is alternately assigned to the effect start operation and theeffect end operation, and a horizontal slide operation on the touchpanel 50 is assigned to the effect switching operation.

Using the touch panel 50 in this way, the space of the user interfacecan be suppressed.

Modified Example 1-4

FIG. 20 is a flowchart illustrating an outline of moving image specialeffect processing performed by an imaging device according to ModifiedExample 1-4 of the first embodiment. Referring to FIG. 20, when a key tostart moving image special effect processing is operated (Yes in stepS61), the image processing control unit 291 set to the moving imagespecial effect flag to on state (step S62). Thereafter, the imagingdevice 1 causes the process to return to the main routine.

However, when the key to start the moving image special effect is notoperated (No in step S61) and then the end instruction is input (Yes instep S63), the image processing control unit 291 performs control suchthat the moving image special effect flag is set to off state (stepS64). Thereafter, the imaging device 1 causes the process to return tothe main routine.

Further, when the key to start the moving image special effect is notoperated (No in step S61), the end instruction is not input (No in stepS63), and then an effect switching instruction is input (Yes in stepS65), the image processing control unit 291 switches a special effect(step S66). Thereafter, the imaging device 1 causes the process toreturn to the main routine.

However, when it is determined in step S65 that the effect switchinginstruction has not been input (No in step S65), the imaging device 1causes the process to return to the main routine.

According to Modified Examples 1-1 to 1-4 of the first embodimentdescribed above, the same effect as in the first embodiment can beobtained.

Second Embodiment

An imaging device according to a second embodiment of the presentinvention is configured such that a function by which a moving imagespecial effect is displayed through a live-view display, the userconfirms a desired effect, and then shooting is performed in a state inwhich a special effect is applied is added to the imaging device of thefirst embodiment.

FIG. 21 is a block diagram illustrating a configuration of an imagingdevice according to the second embodiment.

An imaging device 51 illustrated in FIG. 21 has the same configurationas the imaging device 1 described in the first embodiment except for aninput unit 52. A configuration of the input unit 52 will be describedbelow.

The input unit 52 includes a simulation signal input unit 521, an effectrecording start signal input unit 522, and an effect stop signal inputunit 523 in addition to the setting signal input unit 201 and the movingimage capturing signal input unit 202.

The simulation signal input unit 521 receives an input of a simulationsignal used to apply a moving image special effect only to a live-viewdisplay. The effect recording start signal input unit 522 receives aninput of an effect recording start signal used to record an image towhich the moving image special effect is applied in the SDRAM 25 and therecording medium 23. The effect stop signal input unit 523 receives aninput of an effect stop signal used to stop a moving image specialeffect applied to recording to a live-view display and an image.

FIG. 22 is a diagram illustrating an assignment of a user interfaceaccording to the second embodiment. First, an assignment at the time ofmoving image non-recording will be described. In this case, the exposurecorrection operation is assigned to the up-arrow key 443 of the crosskey 441. The continuous shooting/single shooting operation is assignedto the down-arrow key 444. The AF target operation is assigned to theleft-arrow key 445. The flash setting operation is assigned to theright-arrow key 446. In addition, at the time of moving imagenon-recording, a camera setting operation is assigned to the decisionbutton 442, an AE/AF operation is assigned to a first release operation,and a shooting operation is assigned to a second release operation.

Next, an assignment at the time of moving image recording will bedescribed. In this case, an effect simulation start operation isassigned to the up-arrow key 443. Thus, the up-arrow key 443 serves as apart of the simulation signal input unit 521.

An effect stop operation is assigned to the down-arrow key 444. Thus,the down-arrow key 444 serves as a part of the effect stop signal inputunit 523. Among the moving image special effects, the multiecho and thefluctuation need a stop instruction.

An effect switching operation is assigned to the left-arrow key 445 andthe right-arrow key 446.

An effect recording start function is assigned to the decision button442, and the first release operation and the second release operation ofthe release button 42. Thus, the decision button 442 and the releasebutton 42 serve as a part of the effect recording start signal inputunit 522.

FIG. 23 is a diagram illustrating an outline of a state transition ofthe imaging device 51 at the time of moving image recording. Here, whenthe simulation signal is input through the simulation signal input unit521 in a state (state I) in which a moving image special effect does notremain set to a live-view display and a moving image recording displayafter moving image recording starts, the image processing control unit291 transitions to a state II in which control is performed such that amoving image special effect is applied to a live-view display, and amoving image special effect is not applied during moving imagerecording.

Then, when the effect recording start signal is input through the effectrecording start signal input unit 522 in the state II, the imageprocessing control unit 291 starts to apply a moving image specialeffect even to an image to be recorded in the SDRAM 25 as well as alive-view display (state III).

Then, when the effect stop signal is input through the effect stopsignal input unit 523 in the state III, the image processing controlunit 291 stops applying a moving image special effect to a live-viewdisplay and moving image recording. This causes the imaging device 1 totransit to the state I.

According to the second embodiment of the present invention describedabove, since the imaging device includes the simulation signal inputunit, the effect start signal input unit, and the effect stop signalinput unit, the user can apply a special effect as a trial beforerecording a moving image and check the effect on the screen. Thus, theuser can easily shoot a more creative moving image to which an effectexpected at an intended timing is applied.

Further, in the second embodiment, recording of an image to which amoving image special effect is applied starts at a point in time atwhich an input of the effect recording start signal is received.However, when another operation is not made until a predetermined timeelapses after an input of the simulation signal is received, recordingof an image to which a moving image special effect is applied mayautomatically start.

Modified Example 2-1

FIG. 24 is a diagram illustrating an assignment of a user interface atthe time of moving image recording and at the time of moving imagenon-recording in an imaging device according to Modified Example 2-1 ofthe second embodiment. Referring to FIG. 24, an assignment at the timeof moving image non-recording is the same as the example illustrated inFIG. 22. An assignment at the time of moving image recording will bedescribed below. A one-shot echo start function is assigned to theup-arrow key 443. A transit effect start function is assigned to thedown-arrow key 444. A multi-echo effect simulation start function isassigned to the left-arrow key 445. A fluctuation effect simulationstart function is assigned to the right-arrow key 446. An effectrecording start function and an effect stop function are assigned to thedecision button 442 and the first release operation and the secondrelease operation of the release button 42.

Thus, in Modified Example 2-1, the left-arrow key 445 and theright-arrow key 446 serve as a part of the simulation signal input unit521. Further, in Modified Example 2-1, the decision button 442 and therelease button 42 serve as parts of the effect recording start signalinput unit 522 and the effect stop signal input unit 523.

Modified Example 2-2

In Modified Example 2-2 of the second embodiment, an application stateof the moving image special effect is changed through the touch panel50. FIG. 25 is a diagram illustrating a screen display example in thedisplay unit 21 when the imaging device 51 is in the still imagecapturing standby state. A shooting mode icon I1 (displayed as “P”) isdisplayed on the upper left of the screen Q1. Further, the still imagecapturing menu selection icon I2 (displayed as “STILL MENU”) and amoving image capturing menu selection icon I3 (displayed as “MOVIEMENU”) are displayed on the upper right of the screen Q1. In thefollowing, when a portion of the touch panel 50 corresponding to an icondisplay region is touched, an expression “icon is selected” may be used.

Here, when the menu selection icon I2 is selected in a state in whichthe screen Q1 is displayed, the display control unit 292 causes an iconused to select each of white balance, AF, and photometry and an iconused to return to an original operation display screen to be displayedon the display unit 21 as a still image capturing menu (notillustrated).

However, when the menu selection icon I3 is selected in a state in whichthe screen Q1 is displayed, the display control unit 292 causes an imagerepresenting the moving image capturing standby state to be displayed onthe display unit 21. FIG. 26 is a diagram illustrating a screen displayexample in the display unit 21 when the imaging device 51 is in themoving image capturing standby state. The display unit 21 displays fivetypes of icons including a one-shot echo icon I4 (displayed as “1ShotEcho”), a multiecho icon I5 (displayed as “Multi Echo”), a the transiticon I6 (displayed as “Transit”), a test on icon I7 (displayed as “TESTON”), and a return icon I8 (displayed as “RET”) are displayed on ascreen Q2 illustrated in FIG. 26.

In the moving image capturing standby state illustrated in FIG. 26, anaspect ratio of an image at the time of still image capturing ischanged, and so an image that is more rectangular than a still image isdisplayed. For this reason, the display unit 21 displays various kindsof icons on a non-display area of an image that is of a rectangular bandshape in a vertical direction of a screen.

Among icons displayed on the screen Q2, the one-shot echo icon I4, themultiecho icon I5, and the transit icon I6 are icons used to select theone-shot echo, the multiecho, and the transit as the moving imagespecial effect, respectively. Here, when any one icon is selected, thedisplay unit 21 displays a live-view image to which the selected movingimage special effect has been applied. Thus, the one-shot echo icon I4,the multi-echo icon I5, and the transit icon I6 serve as a part of thesimulation signal input unit 521.

The test on icon I7 is an icon representing a state the selected movingimage special effect is reflected in a live view display but notreflected in moving image recording yet.

The return icon I8 is an icon used to return to an immediately previousimage display.

FIG. 27 is a diagram illustrating a screen display example in thedisplay unit 21 when the transit is selected as the moving image specialeffect in a state in which the screen Q2 is displayed on the displayunit 21. A finish list of switching destinations at the time of transitis selectably displayed on the screen Q3 illustrated in FIG. 27 throughthe display unit 21. A display order of finish items in the finish listdisplay icon I9 can be appropriately changed.

An up-scroll button I91 used to receive an input of an upward scrollsignal and a down-scroll button I92 used to receive an input of adownward scroll signal are disposed on upper and lower ends of thefinish list display icon I9, respectively. For example, when theup-scroll button I91 is selected, a display is scrolled upward, an item“vivid” displayed on the top disappears, an item “fantasic focus” isdisplayed on the top, and another finish item is displayed on thebottom.

In the finish list display icon I9, a currently selected finish item isdisplayed in a manner different from another finish item. Here, adisplay in a different manner is a general term of a displaydistinguished from another finish item, and includes, for example, agray display or a highlight display. FIG. 27 illustrates an example inwhich an item “toy photo” is selected. Here, when a finish item of aswitching destination is selected, the display control unit 292 erasesthe finish list display icon I9. Thereafter, the image processingcontrol unit 291 causes the moving image special effect image processingunit 163 to start transit processing.

Further, when a highlight-displayed item is selected again in the finishlist display icon I9, the process may return to the previously selectedartistic effect. Specifically, let us assume that an effect selectedbefore toy photo is the natural. In this case, when the item “toy photo”is selected again, the process may return to the natural.

FIG. 28 is a diagram illustrating a screen display example after themoving image button 47 is pressed on the screen Q2 illustrated in FIG.26. In addition to the one-shot echo icon I4, the multi-echo icon I5,and the transit icon I6, a decision icon I10 (displayed as “OK”) isdisplayed on the screen Q4 illustrated in FIG. 28. The decision icon I10is an icon used to start recording of an image to which the displayedmoving image special effect is applied. Thus, the decision icon I10serves as a part of the effect recording start signal input unit 522.

Here, when the moving image button 47 is pressed in a state in which thescreen Q4 is displayed and thus moving image recording ends, the displaycontrol unit 292 performs control such that the screen Q2 is displayedon the display unit 21.

FIG. 29 is a diagram illustrating a screen display example in thedisplay unit 21 after the decision icon I10 is selected on the screen Q4illustrated in FIG. 28. In addition to the one-shot echo icon I4, themulti-echo icon 15, and the transit icon I6, an end icon I11 (displayedas “END”) is displayed on a screen Q5 illustrated in FIG. 29. Among theicons, the one-shot echo icon I4 is being highlight-displayed. Thismeans that the one-shot echo is selected.

The end icon I11 is an icon used to input an instruction signal used toend application of the moving image special effect to a moving imagewhich is being recorded. Here, when the end icon I11 is selected in astate in which the screen Q5 is displayed, the image processing controlunit 291 causes the moving image special effect image processing unit163 to end application of the moving image special effect. In this case,the display control unit 292 causes a screen in which the end icon I11is erased from the screen Q5 to be displayed on the display unit 21. Atthis time, it is natural that the moving image special effect is notapplied to a live-view image which is being displayed.

Here, when the moving image button 47 is pressed in a state in which thescreen Q5 is displayed and so moving image recording ends, the displaycontrol unit 292 performs control such that the screen Q2 is displayedon the display unit 21.

FIG. 30 is a diagram illustrating a screen display example in thedisplay unit 21 when the test on icon I7 is selected on the screen Q2illustrated in FIG. 26 and so a test on state is set. A test off iconI12 (displayed as “TEST OFF”) rather than the test on icon I7 isdisplayed on a screen Q6 illustrated in FIG. 30. The test off icon I12is an icon representing a state in which the selected moving imagespecial effect is reflected in a live view display and moving imagerecording.

Here, when the test off icon I12 is selected in a state in which thescreen Q6 is displayed, the display control unit 292 causes the displayunit 21 to display the screen Q2. In other words, when the test off iconI12 is selected, the test on icon I7 is displayed at the position of thetest off icon I12.

Here, when the moving image button 47 is pressed in a state in which thescreen Q6 is displayed on the display unit 21 and so moving imagerecording starts, the display control unit 292 performs control suchthat the screen Q5 is displayed.

According to Modified Examples 2-1 and 2-2 of the second embodimentdescribed above, the same effects as in the second embodiment can beobtained.

Third Embodiment

An imaging device according to a third embodiment of the presentinvention is settable to either of the moving image capturing mode andthe still image capturing mode, and when the moving image capturing modeis set, the user interface is switched to a user interface for movingimage capturing.

FIG. 31 is a block diagram illustrating a configuration of the imagingdevice according to the third embodiment. An imaging device 61illustrated in FIG. 31 has the same configuration as the imaging device1 described in the first embodiment except for an input unit 62. Aconfiguration of the input unit 62 will be described below.

The input unit 62 includes a setting signal input unit 621 that receivesan input of various kinds of signals including a still image capturingcondition setting signal used to set a shooting condition in still imagecapturing and moving image capturing and a moving image capturing signalinput unit 202.

The setting signal input unit 621 includes a mode setting signal inputunit 622 that receives an input of a mode setting signal instructingmode setting. For example, the mode setting signal input unit 622 isimplemented by the mode dial 43.

FIG. 32 is a diagram illustrating a key assignment when the imagingdevice 61 is set to the still image capturing mode or the moving imagecapturing mode. Here, when the imaging device 61 is set to the stillimage capturing mode, the exposure correction operation is assigned tothe up-arrow key 443. The continuous shooting/single shooting operationis assigned to the down-arrow key 444. The AF target operation isassigned to the left-arrow key 445. The flash setting operation isassigned to the right-arrow key 446. The playback operation is assignedto the playback button 46. The erase operation is assigned to the erasebutton 49.

Next, an example in which the imaging device 61 is set to the movingimage capturing mode will be described. The up-arrow key 443 and theleft-arrow key 445 are assigned the same operations to when the stillimage mode is set, that is, the exposure correction operation and the AFtarget operation, respectively. The down-arrow key 444 is assigned thetransit effect start operation. The right-arrow key 446 is assigned thefluctuation effect start/stop operation. The playback button 46 isassigned the one-shot echo start operation. The erase button 49 isassigned the multi-echo start/stop operation.

In the third embodiment, a stop setting as well as a start setting canbe arbitrarily made on the fluctuation effect. However, when thefluctuation effect stops when a predetermined time elapses after theeffect starts similarly to the first embodiment, it is preferable thatthe fluctuation effect start operation be assigned to the right-arrowkey 446.

FIGS. 33A and 33B are flowcharts illustrating an outline of processingperformed by the imaging device 61. The processes of steps S71 to S76 inFIG. 33A sequentially correspond to the processes of steps S1 to S6 inFIG. 13A.

Here, when it is determined in step S77 that the moving image button 47has been operated (Yes in step S77), the control unit 29 inverts therecording flag representing that moving image recording is beingperformed (step S78).

Next, the control unit 29 determines whether or not the recording flagrecorded in the SDRAM 25 is in the on state (step S79). Here, when it isdetermined that the recording flag is in the on state (Yes in step S79),the control unit 29 generates a moving image file used to record imagedata in the recording medium 23 in time series and stores the movingimage file in the recording medium 23 (step S80). Thereafter, theimaging device 61 causes the process to proceed to step S81. However,when it is determined in step S79 that the recording flag is not in theon state (No in step S79), the imaging device causes the process toproceed to step S81.

Then, when it is determined in step S81 that a mode setting has beenchanged (Yes in step S81) and then it is determined that the movingimage mode remains set (Yes in step S82), the control unit 29 performs aprocess of setting a user interface for a special moving image (stepS83). Through this setting, for example, an assignment for the movingimage mode illustrated in FIG. 32 is executed. As a result, the controlunit 29 recognizes a signal subsequently received by the user interfacewhich is the setting target in step S83 as a signal for a special movingimage with reference to FIG. 32.

Meanwhile, when it is determined in step S81 that a mode setting hasbeen changed (Yes in step S81) and then it is determined that the stillimage mode remains set (No in step S82), the control unit 29 performs aprocess of setting a user interface for a still image (step S85). Atthis time, a key assignment for the still image mode illustrated in FIG.32 is executed.

The processes of steps S86 to S100 sequentially correspond to theprocesses of steps S14 to S28 described in the first embodiment.

According to the third embodiment of the present invention describedabove, when the moving image capturing mode is set, the setting signalinput unit switches the user interface to the user interface for themoving image special effect, and the control unit recognizes that themoving image special effect setting signal has been input. Thus, theuser can easily apply a special effect to a moving image by performing asimple operation at the time of shooting. The setting of the userinterface is switched according to the mode setting, and shooting isperformed after the moving image special effect is checked through thelive-view image.

Further, according to the third embodiment, the user interfaceconfiguring the setting signal input unit can be provided with both aninput function for a moving image and an input function for a stillimage. Thus, the number of user interfaces can be reduced. Thus, theuser interface is suitable for device miniaturization.

In addition, according to the third embodiment, the user can clearlyunderstand the application of the moving image special effect before themoving image is recorded in the moving image capturing standby state orthe still image capturing standby state. Thus, the user can cause amoving image with a special effect to be captured and recorded as amoving image to be imaged in advance or can consider a timing to apply aspecial effect in advance.

Modified Example 3-1

FIG. 34 is a diagram illustrating a key assignment when an imagingdevice according to Modified Example 3-1 of the third embodiment is setto the still image capturing mode or the moving image capturing mode.Here, when the imaging device according to Modified Example 3-1 is setto the still image capturing mode, a key assignment of the cross key 441is the same as in the third embodiment.

An example of a key assignment when the imaging device according toModified Example 3-1 is set to the moving image capturing mode will bedescribed below. The up-arrow key 443 is assigned the one-shot echostart operation. The down-arrow key 444 is assigned the transit startoperation. The left-arrow key 445 is assigned the multi-echo start/stopoperation. The right-arrow key 446 is assigned the fluctuationstart/stop operation.

Modified Example 3-2

FIG. 35 is a diagram illustrating a screen display example in thedisplay unit 21 when an imaging device according to Modified Example 3-2of the third embodiment is in the still image capturing standby state.In Modified Example 3-2, various kinds of signals are input using thetouch panel 50. In addition to the still image capturing menu selectionicon I2, a shooting mode icon I1′ is displayed on a screen Q7illustrated in FIG. 35. The shooting mode icon I1′ is configured toreceive a touch input and serves as a part of the mode setting signalinput unit 622.

FIG. 36 is a diagram illustrating a screen display example in thedisplay unit 21 when the shooting mode icon I1′ is selected on thescreen Q7. Here, six kinds of icons including a P mode icon I13(displayed as “P”), an A mode icon I14 (displayed as “A”), an S modeicon I15 (displayed as “S”), an M mode icon I16 (displayed as “M”), amoving image icon I17 (displayed as “MOVIE”), and a decision icon 110are displayed on a lower portion of a screen Q8 illustrated in FIG. 36.Among the icons, the P mode icon 113, the A mode icon I14, the S modeicon I15, and the M mode icon I16 are icons used to set the shootingmode (exposure mode), and have the same function as the mode dial 43.

Here, the moving image icon I17 is selected on the screen Q8, theimaging device 1 is set to the moving image mode and becomes the movingimage capturing standby state.

FIG. 37 is a diagram illustrating a screen display example in thedisplay unit 21 in the moving image capturing standby state. Theone-shot echo icon I4, the multi-echo icon I5, the transit icon I6, andthe return icon I8 are displayed on a screen Q9 illustrated in FIG. 37.

Here, when any one of the one-shot echo icon I4, the multi-echo icon I5,and the transit icon I6 is selected in the state illustrated in FIG. 37,the display control unit 292 applies a special effect corresponding tothe selected moving image special effect and displays a live-view image.Here, when the transit icon I6 is selected, the screen Q3 illustrated inFIG. 27 is displayed. Here, when the moving image button 47 is pressedin a state in which the moving image special effect is selected, thecontrol unit 29 starts recording of a moving image to which the movingimage special effect currently displayed through the live-view image isapplied.

According to Modified Examples 3-1 and 3-2 of the third embodimentdescribed above, the same effects as in the third embodiment can beobtained.

Fourth Embodiment

In a fourth embodiment of the present invention, an imaging devicecapable of capturing a moving image can set a moving image specialeffect applying mode in which shooting is performed in a state in whicha moving image special effect is applied as one of the shooting modes.

FIGS. 38A and 38B are flowcharts illustrating an outline of processingperformed by an imaging device according to the fourth embodiment of thepresent invention. The imaging device according to the fourth embodimenthas the same configuration as the imaging device 61 described in thethird embodiment. The processes of steps S111 to S114 and S116 in FIG.38A sequentially correspond to the processes of steps S71 to S74 and S76in FIG. 33A.

In step S115, camera setting is performed (step S115). Here, the camerasetting includes selection of a special effect in addition to finishsetting, still image recording mode setting, moving image recording modespecial effect setting, transit switching destination setting, andfluctuation effect setting. Thus, in the fourth embodiment, the movingimage special effect is assumed to be selected in advance. The specialeffect may be selected through the mode dial 43.

Here, when it is determined in step S117 that the moving image button 47has been operated (Yes in step S117), the control unit 29 inverts therecording flag representing that moving image recording is beingperformed (step S118).

Next, the control unit 29 determines whether or not the recording flagrecorded in the SDRAM 25 is in the on state (step S119). Here, when itis determined that the recording flag is in the on state (Yes in stepS119), the control unit 29 generates a moving image file used to recordimage data in the recording medium 23 in time series and stores themoving image file in the recording medium 23 (step S120). Thereafter,the imaging device 61 causes the process to proceed to step S121.However, when it is determined in step S119 that the recording flag isnot in the on state (No in step S119), the moving image special effectimage processing unit 163 causes the process to proceed to step S121.

In step S121, the control unit 29 determines whether or not the imagingdevice 61 remains set to the moving image special effect mode. Here,when it is determined that the imaging device 61 remains se to themoving image special effect mode (Yes in step S121), the control unit 29sets the moving image special effect flag to the on state (step S122).However, when the control unit 29 determines in step S121 that theimaging device 61 does not remain set to the moving image special effectmode (No in step S121), the control unit 29 sets the moving imagespecial effect flag to the off state (step S123).

The processes of steps S124 to S136 sequentially correspond to theprocesses of steps S88 to S100 described in the third embodiment.

According to the fourth embodiment of the present invention describedabove, the moving image special effect mode can be set, and thus bychanging the mode, a moving image to which the moving image specialeffect is applied can be simply captured.

Further, in the imaging device according to the fourth embodiment, it ispreferable that the still image mode can be set.

Fifth Embodiment

In a fifth embodiment of the present invention, even when a still imageis captured during moving image capturing, a moving image special effectcan be applied based on previously recorded moving image data.

FIG. 39 is a block diagram illustrating a configuration of an imagingdevice according to the fifth embodiment. An imaging device 71illustrated in FIG. 39 has the same configuration as the imaging device1 described in the first embodiment except for an image processing unit72. A configuration of the image processing unit 72 will be describedbelow.

The image processing unit 72 includes a basic image processing unit 161,an artistic effect image processing unit 162, and a moving image specialeffect image processing unit 721. The moving image special effect imageprocessing unit 721 includes a resizing processing unit 722 thatinterpolates the number of pixels of moving image data and resizes thenumber of pixels of moving image data.

FIGS. 40A and 40B are flowcharts illustrating the details of imageprocessing performed by the imaging device 71. Except for imageprocessing, an outline of processing performed by the imaging device 71is the same as in the first embodiment (see FIGS. 13A and 13B).Referring to FIGS. 40A and 40B, the basic image processing unit 161performs basic image processing (step S141), and the artistic effectimage processing unit 162 performs artistic effect image processing(step S142). Here, the basic image processing and the artistic effectimage processing are the same as described in the first embodiment.

Thereafter, the control unit 29 determines whether or not the movingimage special effect remains set (step S143). Here, when it isdetermined that the moving image special effect remains set (Yes in stepS143), the imaging device 71 causes the process to proceed to step S144.However, when it is determined that the moving image special effect doesnot remain set (No in step S143), the imaging device 71 causes theprocess to return to the main routine (see FIGS. 13A and 13B).

In step S144, the control unit 29 determines whether or not the setspecial effect is the multi-echo. Here, when it is determined that theset special effect is the multi-echo (Yes in step S144), the imagingdevice 71 causes the process to proceed to step S145. However, when itis determined that the set special effect is not the multi-echo (No instep S144), the imaging device 71 causes the process to proceed to stepS149 which will be described later.

Here, when it is determined in step S145 that the imaging device 71remains set to the still image mode (Yes in step S145), the resizingprocessing unit 722 resizes a previous moving image frame (step S146).This process is performed because a moving image and a still imagediffer in the number of pixels of image data.

FIG. 41 is a diagram schematically illustrating an outline of resizingprocessing performed by the resizing processing unit 722. Referring toFIG. 41, the size of a frame corresponds to the number of pixels ofimage data. Here, when resizing processing is performed on moving imagedata, the resizing processing unit 722 generates a resized frame 301′ byincreasing the number of pixels by an interpolation such that the numberof pixels of a moving image frame 301 in a horizontal direction becomesalmost equal to the number of pixels of a still image frame 302 in ahorizontal direction in a state in which the aspect ratio of the movingimage frame 301 is maintained.

Next, the moving image special effect image processing unit 721synthesizes the resized moving image frame with the still image frame(step S147). Specifically, the moving image special effect imageprocessing unit 721 generates a synthesized image signal by adding asignal of the moving image frame to a signal of the still image frame ata predetermined ratio for each color data of a pixel. Here, when a ratioof a previous frame image signal is c (<1), the synthesized image signalis represented by:

c×previous frame image signal+(1−c)×still image frame signal.

FIG. 42 is a diagram for schematically describing synthesis processingin the still image mode and the synthesis ratio c the previous frameimage signal. Referring to FIG. 42, the ratio c of an image signal whichis a previous frame has an almost constant value c₀ in a region D1 inwhich an image shared between a resized previous frame and a still imageis present. The constant value c₀ has a value smaller than, for example,0.5. On the other hand, near the boundary between a region D2 in which amoving image frame signal is not present an the region D1, the ratio csteadily decreases from the region D1 toward the region D2, and theratio c becomes zero (0) in the region D2.

Here, when it is determined in step S145 that the imaging device 71remains set to the moving image mode (No in step S145), the moving imagespecial effect image processing unit 721 synthesizes the previous framewith the moving image frame as is (step S148).

Subsequently to step S147 or S148, the control unit 29 determineswhether or not an effect to apply is the one-shot echo (step S149).Here, when it is determined that an effect to apply is the one-shot echo(Yes in step S149) and then it is determined that the still imagecapturing mode remains set (Yes in step S150), the moving image specialeffect image processing unit 721 resizes the previous frame (step S151).However, when it is determined that an effect to apply is the one-shotecho (Yes in step S149) and then it is determined that the still imagecapturing mode does not remain set (No in step S150), the imaging device71 causes the process to proceed to step S152.

Thereafter, the moving image special effect image processing unit 721synthesizes the previous frame with a current frame at a predeterminedsynthesis ratio (step S152). It is preferable that the synthesis ratiobe set such that the ratio of the current frame is higher.

Meanwhile, when it is determined in step S149 that a moving imagespecial effect to apply is not the one-shot echo (No in step S149), theimaging device 71 causes the process to proceed to step S153.

In step S153, the control unit 29 determines whether or not an effect toapply is a fluctuation. Here, when it is determined that an effect toapply is the fluctuation (Yes in step S153), the moving image specialeffect image processing unit 721 adds the fluctuation (step S154).However, when it is determined that an effect to apply is not thefluctuation (No in step S153), the imaging device 71 causes the processto proceed to step S155.

In step S155, the control unit 29 determined whether or not an effect toapply is the transit (step S155). Here, when it is determined that aneffect to apply is the transit (Yes in step S155) and then the imagingdevice 71 is not in a still image capturing state (No in step S156), thebasic image processing unit 161 and the artistic effect image processingunit 162 execute basic image processing (step S157) and artistic effectimage processing (step S158) according to settings of finish andswitching destinations, respectively.

Thereafter, the moving image special effect image processing unit 721performs synthesis processing of two images for the transit (step S159).After step S159, the imaging device 71 causes the process to return tothe main routine.

Meanwhile, when it is determined in step S155 that an effect to apply isnot the transit (No in step S155), the imaging device 71 causes theprocess to return to the main routine without applying the transit tothe still image.

Further, when it is determined in step S156 that the imaging device 71is in the still image capturing state (Yes in step S156), the imagingdevice 71 causes the process to return to the main routine.

According to the fifth embodiment of the present invention describedabove, a special effect such as the multi-echo or the one-shot echo isapplied to a still image, but in this case, since resizing processing isperformed, even though a moving image is synthesized with a still image,a feeling of strangeness does not occur. Thus, according to the fifthembodiment, appropriate residual image effects can be applied to amoving image and a still image while making use of theircharacteristics.

Further, in the fifth embodiment, both a moving image and a still imageare synthesized with a previous frame, but a synthesized image may begenerated using a plurality of previous frames. In this case, it is morepreferable that a residual image effect be increased using a largernumber of previous frames in a still image than in a moving image.

Modified Example 5-1

FIG. 43 is a diagram schematically illustrating an outline of resizingprocessing performed by an imaging device according to Modified Example5-1 of the fifth embodiment. Even in FIG. 43, the size of a framecorresponds to the number of pixels of image data. In Modified Example5-1, when image data is recorded, even in case of a moving image, imagedata 303 having the same aspect ratio as a still image is recorded inthe SDRAM 25. A portion of the image data 303 having an aspect ratio ofa moving image is compressed and then recorded in the recording medium23 as moving image data 304.

In Modified Example 5-1, when a still image is captured, the movingimage special effect image processing unit 721 performs the resizingprocess using the image data 303 stored in the SDRAM 25 as the previousframe and so generates resized image data 303′ which is equal in thesame number of pixels to the still image frame. Thus, even when a stillimage data is synthesized with a moving image data, image data with thesame aspect ratio can be synthesized.

According to Modified Example 5-1 of the fifth embodiment describedabove, the synthesis process can be performed in a state in which anaspect ratio of a previous frame matches an aspect ratio of a stillimage. Thus, among subjects shown in a still image frame, a subjectshown outside a region of a moving image aspect ratio of a previousframe image can have a natural residual effect.

Sixth Embodiment

In a sixth embodiment of the present invention, similarly to the fifthembodiment, even when a still image is captured during moving imagecapturing, a moving image special effect is applied using moving imagedata acquired before the still image in terms of time, and even a stillimage to which a moving image special effect is not applied is recorded.An imaging device according to the sixth embodiment has the sameconfiguration as the imaging device 71 described in the fifthembodiment.

FIGS. 44A and 44B are flowcharts illustrating the details of imageprocessing performed by the imaging device 71 according to the sixthembodiment. Referring to FIGS. 44A and 44B, the basic image processingunit 161 performs basic image processing (step S161). The artisticeffect image processing unit 162 performs artistic effect imageprocessing (step S162).

Thereafter, the control unit 29 determines whether or not a moving imagespecial effect remains set (step S163). Here, when it is determined thatthe moving image special effect remains set (Yes in step S163), theimaging device 71 causes the process to proceed to step S164. However,when it is determined that the moving image special effect does notremain set (No in step S163), the imaging device 71 causes the processto return to the main routine.

Here, when it is determined in step S164 that the imaging device 1remains set to the still image capturing mode (Yes in step S164), thecontrol unit 29 performs control such that a still image is recorded inthe recording medium 23 (step S165). However, when it is determined thatthe imaging device 1 remains set to the still image capturing mode (Noin step S164), the imaging device 1 causes the process to proceed tostep S166.

The processes of steps S166 to S181 sequentially correspond to theprocesses of steps S144 to S159 described in the fifth embodiment (seeFIGS. 40A and 40B).

According to the sixth embodiment of the present invention describedabove, a special effect such as the multi-echo or the one-shot echo isapplied even to a still image. However, in this case, resizingprocessing is performed, and thus even though a moving image issynthesized with a still image, a feeling of strangeness does not occur.Thus, according to the sixth embodiment, appropriate residual imageeffects can be applied to a moving image and a still image while makinguse of their characteristics.

Further, according to the sixth embodiment, a still image to which amoving image special effect is desired to be applied is automaticallyrecorded, and thus the user can select his/her preferred image whenplaying back a still image.

Seventh Embodiment

FIG. 45 is a diagram schematically illustrating an outline of imageprocessing performed by an imaging device according to a seventhembodiment of the present invention. In the seventh embodiment, when themulti-echo is applied as the moving image special effect, a moving imageand a still image differ in the number of pieces of moving image data ofa previous frame which is a synthesis target. Specifically, in case of amoving image, three pieces of moving image data are synthesized, whereasin case of still image, five pieces of moving image data aresynthesized. The imaging device according to the seventh embodiment hasthe same configuration as the imaging device 71 described in the fifthembodiment.

FIGS. 46A and 46B are flowcharts illustrating an outline of imageprocessing performed by the imaging device 71 according to the seventhembodiment. Referring to FIGS. 46A and 46B, first, the control unit 29manages RAW image data (Bayer data) acquired by the imaging element 12through a ring buffer secured inside the SDRAM 25 (step S191).

FIG. 47 is a diagram illustrating a relation between Bayer data and anexposure amount. As can be seen from a curved line L_(k) illustrated inFIG. 47, a numerical value of the Bayer data has a knee characteristicin which rising is steep in a dark portion, but rising is gentle in abright portion. The curved line L_(k) having this characteristic isobtained by finely performing a quantization step as it is closer to thebright portion and roughly performing the quantization step as it iscloser to the dark portion. In this way, in the seventh embodiment,Bayer data is quantized by a quantization step different from a normalquantization step which is constant regardless of an exposure amount.Thus, a numerical value characteristic of Bayer data is close to a gammacharacteristic, influence on an image quality is reduced, and data iseasily compressed. As a result, a small amount of image data can bestored in the SDRAM 25.

FIG. 48 is a diagram schematically illustrating an output line of a datastructure stored in the SDRAM 25. In data structure 401 illustrated inFIG. 48, buffers Bu1 to Bu5 which are equal in number to the number(five in FIG. 48) used at the time of image synthesis are secured asring buffers. The buffer Bui (i=1 to 5) includes a region Bui1 used tostore Bayer data and a region Bui2 used to store an address of a nextbuffer. In the following, an address of a buffer Bui is denoted by Adi.

First, when Bayer data is stored, initial storage destinationinformation is set to “storage destination address=Ad1, endaddress=NULL(0).” Thereafter, when Bayer data is stored, the storagedestination information is set to “storage destination address=Ad2, endaddress=Ad1.” This means that data is present in ascending order by timefrom the buffer of the address Ad1 to the immediately previous buffer ofthe address Ad2. In other words, it means that single Bayer data remainsstored.

Thereafter, when second Bayer data is stored, the storage destinationinformation is set to “storage destination address=Ad3, endaddress=Ad1.” In this case, Bayer data is stored in the two buffers Bu1and Bu2.

Thereafter, the same process as described above is repeated, and thenwhen fifth Bayer data is stored, the storage destination information isset to “storage destination address=Ad1, end address=Ad1.” In this case,Bayer data is stored in the five buffers Bu1 to Bu5.

Thereafter, when Bayer data is continuously stored, since the oldestBayer data is stored in the buffer Bu1, the Bayer data is updated tolatest data, and Bayer data stored in the buffer Bu1 becomes the oldestdata. Thus, the storage destination information is set to “storagedestination address=Ad2, end address=Ad2.”

Thereafter, by repeating the same process as described above, a constantnumber of pieces of Bayer data are managed in the SDRAM 25 whilereplacing the oldest Bayer data with latest Bayer data.

Referring back to FIGS. 46A and 46B, step S192 and subsequent processeswill be described. The basic image processing unit 161 performs basicimage processing (step S192). The artistic effect image processing unit162 performs artistic effect image processing (step S193). The imagedata (Bayer data) which have been subjected to basic image processingand artistic effect image processing is stored in the buffer of theSDRAM 25.

Thereafter, the control unit 29 determines whether or not the movingimage special effect remains set (step S194). Here, when it isdetermined that the moving image special effect remains set (Yes in stepS194), the imaging device 71 causes the process to proceed to step S195.However, when it is determined that the moving image special effect doesnot remain set (No in step S194), the imaging device 71 causes theprocess to return to the main routine.

Here, when it is determined in step S195 that the multi-echo remains setas the moving image special effect (Yes in step S195), the imagingdevice 71 performs multi-echo processing (step S196). The details ofmulti-echo processing will be described later.

The processes of steps S197 to S207 sequentially correspond to theprocesses of steps S171 to S181 of FIGS. 44A and 44B.

FIG. 49 is a flowchart illustrating an outline of multi-echo processing.Referring to FIG. 49, the moving image special effect image processingunit 721 multiplies an acquired image signal by a predeterminedcoefficient (gain) for each pixel (step S211). A value of thiscoefficient is d₀ which will be described later.

Thereafter, the control unit 29 sets a value of a counter I representinga repeat count to zero (0) (step S212).

Next, the moving image special effect image processing unit 721 acquiredata corresponding to a (I+1)-th previous frame (step S213), and thenperforms a linear transform (step S214). Here, the linear transform isto eliminate a difference in characteristic at the time of quantizationstep of each frame, and has a characteristic (a deknee) characteristic)opposite to the characteristic in the above-described quantization step.As the linear transform is performed, Bayer data has a numerical valueproportional to an exposure amount.

Thereafter, the basic image processing unit 161 performs basic imageprocessing on the image data which has been subjected to the lineartransform (step S215). Further, the artistic effect image processingunit 162 performs artistic effect image processing on the image datawhich has been subjected to basic image processing in step S215 (stepS216).

The process subsequent to step S216 depends on whether an image acquiredby the imaging device 71 is a still image or a moving image. First, anexample in which the acquired image is a still image (Yes in step S217)will be described. In this case, the resizing processing unit 722resizes the image data (step S218), and then performs the synthesisprocess using the resized image data (step S219). Here, the synthesisprocessing is represented by:

frame buffer image data+d×image data.

FIG. 50 is a diagram illustrating a relation between a value of an imagegain d and a repeat count I. In an example illustrated in FIG. 50, asthe repeat count I increases, the image gain d decreases, and a settingis made such that a value obtained by adding coefficients correspondingto all possible repeat counts I by all repeat counts I is 1.0(d₀+d₁+d₂+d₃+d₄=1.0). Here, when I is zero, the image gain d₀ has avalue of an image gain applied in step S211 described above.

Next, the control unit 29 increases the counter I by one (1) to be setto (I+1) (step S220). Thereafter, when the counter I is a predeterminedvalue I₀ (here, I₀=5) (Yes in step S221), the imaging device 71 causesthe process to return to step S213. However, when the counter I is equalto or more than the predetermined value I₀ (No in step S221), theimaging device 71 causes the process to return to the main routine.

Next, an example in which the acquired image is a moving image (No instep S217) will be described. In this case, the moving image specialeffect image processing unit 721 synthesizes two frames (step S222).Here, similarly to a still image, the synthesis processing isrepresented by:

frame buffer image data+e×image data

FIG. 51 is a diagram illustrating a relation between a value of an imagegain e applied when the acquired image is a moving image and the repeatcount I. In the example illustrated in FIG. 51, similarly to the case ofa still image, as the repeat count I increases, the coefficientdecreases, and a setting is made such that a value obtained by addingcoefficients corresponding to all possible repeat counts I by all repeatcounts I is 1.0 (e₀+e₂+e₄=1.0). However, since a moving image is largerin an increment amount than a still image, when a moving image and astill image are assumed to have the same I value, it is preferable thata coefficient applied to a moving image be larger than a coefficientapplied to a still image. Further, in order to cause an image gain valueof a still image to be equal to an image gain value of a moving image atstep S211, d₀=e₀ needs to be satisfied.

Next, the control unit 29 increases the counter I by two (2) to be setto (I+2) (step S223). Thereafter, the imaging device 1 causes theprocess to proceed to step S221.

According to the seventh embodiment of the present invention describedabove, an appropriate residual image effect can be implemented on eachof a moving image and a still image, and an image can be generated byminimum processing.

Further, according to the seventh embodiment, since a characteristic ofa coefficient can be arbitrarily set, a degree of freedom of a design islarge.

Further, in the seventh embodiment, settings such as the image gains dand e, the number of ring buffers, and interval between frames used fora synthesis may be changed by an operation input from the lens operatingunit 35 or the like. By changing various settings in this way, theuser's preferred residual image effect can be generated.

In the seventh embodiment, a form in which a ring buffer is managed isnot limited to the above-described method. For example, a ring buffercan be managed using a queue. Next, a method of managing a ring bufferusing a queue will be described.

FIG. 52 is a diagram schematically illustrating a main part of a datastructure stored in the SDRAM 25. In the data structure 402 illustratedin FIG. 52, buffers bu1 to bu5 which are equal in number to the number(five in FIG. 52) used at the time of image synthesis are secured asring buffers. The buffer bui (i=1 to 5) stores Bayer data.

FIG. 53 is a diagram illustrating a formed in which a ring buffer ismanaged using a queue. First, when an element is present in an emptyqueue, an element at the head is fetched, corresponding Bayer data isstored in the fetched element, and then the element is added to the tailof a use queue. In the example of FIG. 53( a), a buffer but at the headof the empty queue is fetched, corresponding Bayer data1 is stored inthe buffer bu1, and then the buffer bu1 is added to the tail of the usequeue (FIG. 53( b)). As this process is repeated, in the exampleillustrated in FIG. 53( a), the empty queue becomes empty when fifthstorage is completed, and all buffers are added to the use queue.Thereafter, Bayer data is continuously stored while moving the buffersfrom the head to the tail of the use queue (FIG. 53( d)).

In this way, any well-known technique can be applied in order to managemoving image data using a ring buffer.

Further, in the seventh embodiment, an extraction interval of image datato be synthesized may be changed, and the number of pieces of image datato be synthesized may be changed. Further, in the seventh embodiment,the number of pieces of Bayer data stored as a ring buffer may bechanged. This setting change can be made in response to a setting signalinput through the setting signal input unit 201.

Other Embodiments

The embodiments for embodying the present invention have been describedso far, but the present invention is not limited to the first to seventhembodiments.

For example, in the present invention, an electronic view finder may bedisposed in the main body separately from a display unit, and thepresent invention may be applied to the electronic view finder. In thiscase, it is preferable that the display unit and the electronic viewfinder differ from each other in a method of viewing a moving imagespecial effect.

Further, in the present invention, the lens unit may be formedintegrally with the main body.

In addition, the imaging device according to the present invention canbe applied to electronic devices such as digital cameras with anaccessory mounted thereto, digital video cameras, portable phones with ashooting function, and tablet type portable devices as well as digitalsingle-lens reflex cameras.

Furthermore, in the description of the flowchart in this disclosure, asequence relation of processing between steps is specified using anexpression such as “first,” “thereafter,” and “next.” However, asequence of processing necessary to embody the present invention is notuniquely decided. In other words, a sequence of processing in aflowchart described in this disclosure may be changed within aconsistent range.

As described above, the present invention can include variousembodiments which have not been described herein, and various designchanges can be made within the scope of a technical spirit set forth inclaims.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

What is claimed is:
 1. An imaging device that captures a subject,generates image data of the subject, captures a moving image, andcaptures a still image during moving image capturing including directlyafter moving image capturing end, the device comprising: a moving imagespecial effect image processing unit that performs image processing ofapplying a first special effect causing a visual effect over a pluralityof frames corresponding to image data of a moving image, and imageprocessing of applying a second special effect corresponding to thefirst special effect to a still image captured during moving imagecapturing using image data of a moving image captured before the stillimage; and a control unit that controls an application form of the firstand second special effects in the moving image special effect imageprocessing unit.
 2. The imaging device according to claim 1, wherein thecontrol unit performs control such that the second special effect is notapplied to a still image included between the plurality of frames towhich the first special effect is applied.
 3. The imaging deviceaccording to claim 1, wherein the control unit performs control suchthat the second special effect is applied to a still image includedbetween the plurality of frames to which the first special effect isapplied.
 4. The imaging device according to claim 1, further comprising:a storage unit that stores the image data, wherein the control unitperforms control such that the second special effect is applied to astill image included between the plurality of frames to which the firstspecial effect is applied, and stores image data to which the secondspecial effect is applied and image data to which the second specialeffect is not applied in the storage unit.
 5. The imaging deviceaccording to claim 1, wherein the moving image special effect imageprocessing unit includes a resizing processing unit that performsresizing processing of approximating the number of pixels of image dataof a moving image captured before the still image to the number ofpixels of image data of the still image, and the moving image specialeffect image processing unit synthesizes the image data of a movingimage which has been subjected to resizing processing by the resizingprocessing unit with the image data of the still image.
 6. The imagingdevice according to claim 5, wherein image processing performed by themoving image special effect image processing unit includes multi-echoprocessing of continuously performing processing of synthesizing imagedata of an immediately previous frame with most recent image data at apredetermined ratio, and when the most recent image data is image dataof a still image, the resizing processing unit performs resizingprocessing on the image data.
 7. The imaging device according to claim5, wherein image processing performed by the moving image special effectimage processing unit includes one-shot echo processing in whichspecific image data is sequentially synthesized with image data capturedafter the specific image data at a predetermined ratio, and a synthesisratio of the specific image data decreases over time, and when the mostrecent image data is image data of a still image, the resizingprocessing unit performs resizing processing on the image data.
 8. Animaging device that captures a subject, generates image data of thesubject, captures a moving image, and captures a still image duringmoving image capturing including directly after moving image capturingend, the device comprising: a moving image special effect imageprocessing unit that performs moving image special effect imageprocessing of synthesizing captured image data with a plurality of imagedata captured before the image data at a predetermined ratio; and acontrol unit that changes the number of image data synthesized by themoving image special effect image processing unit depending on whetherthe captured image data is a moving image or a still image.
 9. Theimaging device according to claim 8, wherein the number of image datasynthesized by the moving image special effect image processing unitwhen the captured image data is a moving image is larger than when thecaptured image data is a still image.
 10. The imaging device accordingto claim 8, wherein the earlier image data is captured in terms of time,the smaller the predetermined ratio is.
 11. The imaging device accordingto claim 8, wherein the moving image special effect image processingunit includes a resizing processing unit that performs resizingprocessing of approximating the number of pixels of image data of amoving image captured before the still image to the number of pixels ofimage data of the still image, and the moving image special effect imageprocessing unit synthesizes the image data of a moving image which hasbeen subjected to resizing processing by the resizing processing unitwith the image data of the still image.
 12. The imaging device accordingto claim 11, wherein image processing performed by the moving imagespecial effect image processing unit includes multi-echo processing ofcontinuously performing processing of synthesizing image data of animmediately previous frame with most recent image data at apredetermined ratio, and when the most recent image data is image dataof a still image, the resizing processing unit performs resizingprocessing on the image data.
 13. The imaging device according to claim12, further comprising a setting signal input unit that receives aninput of a setting signal used to set the ratio.
 14. The imaging deviceaccording to claim 11, wherein image processing performed by the movingimage special effect image processing unit includes one-shot echoprocessing in which specific image data is sequentially synthesized withimage data captured after the specific image data at a predeterminedratio, and a ratio of the specific image data decreases over time, andwhen the most recent image data is image data of a still image, theresizing processing unit performs resizing processing on the image data.15. The imaging device according to claim 14, further comprising asetting signal input unit that receives an input of a setting signalused to set the ratio.
 16. The imaging device according to claim 8,wherein the image data is RAW data which has not been subjected to acompression coding process.
 17. The imaging device according to claim16, further comprising: a temporary storage unit that temporarily storesthe RAW data, wherein the moving image special effect image processingunit synthesizes a plurality of RAW data stored in the temporary storageunit.
 18. The imaging device according to claim 17, wherein an aspectratio of a screen in still image capturing is equal to an aspect ratioof a screen in moving image capturing, and an image with the aspectratio is temporarily stored in the temporary storage unit at the time ofmoving image capturing.
 19. An imaging method performed by an imagingdevice that captures a subject, generates image data of the subject,captures a moving image, and captures a still image during moving imagecapturing including directly after moving image capturing end, themethod comprising: performing image processing of applying a firstspecial effect causing a visual effect over a plurality of framescorresponding to image data of a moving image, and image processing ofapplying a second special effect corresponding to the first specialeffect to a still image captured during moving image capturing usingimage data of a moving image captured before the still image.
 20. Animaging method performed by an imaging device that captures a subject,generates image data of the subject, captures a moving image, andcaptures a still image during moving image capturing including directlyafter moving image capturing end, the method comprising: performingmoving image special effect image processing of synthesizing capturedimage data with a plurality of image data captured before the image dataat a predetermined ratio; and changing the number of image datasynthesized by the moving image special effect image processing unitdepending on whether the captured image data is a moving image or astill image.
 21. An imaging device that captures a subject, generatesimage data of the subject, and captures a moving image and a stillimage, the device comprising: a moving image special effect imageprocessing unit that applies an expression of temporally changing avisual effect over a plurality of frames; and an artistic effectprocessing unit that applies an artistic effect causing a visual effectto one image data, wherein the moving image special effect imageprocessing unit performs transit processing of steadily changing anartistic effect over a plurality of frames, the transit processingsynthesizing an image obtained by multiplying a first artistic effectimage by a coefficient b changing over time with an image obtained bymultiplying a second artistic effect image different from the firstartistic effect image by a coefficient 1−b, and the coefficient b ischanged in response to a start trigger input of the transit processing.22. The imaging device according to claim 21, wherein after the starttrigger input of the transit processing, the coefficient b smoothlydecreases over time and becomes zero (0) when a predetermined timeelapses.